Incident Library
Applied filters Root causeDependency failureSymptomDNS resolution failuresOrganisationOctez ConnectClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0269 Octez Connect services went down for hours when its only DNS provider failed and the nodes' records stopped resolving
    Octez Connect2025
    Symptoms

    On 5 February 2025 at 04:34 the monitoring system flagged DNS resolution problems on the first Octez Connect node and raised a Pager Duty alert. Every server that relied on the same DNS provider went down, and users could not reach the services until the provider restored DNS.

    DNS resolution failuresService outage
    Trigger

    An outage at Octez Connect's DNS provider, metanet.ch, stopped the DNS records of its nodes from resolving.

    Root cause
    Dependency failure
    Component
    Authoritative DNS servers and records
    Contributing factor
    Single point of failureLimited debug access
    Detection

    The monitoring system flagged DNS resolution problems on the first node at 04:34 and raised a Pager Duty alert.

    Remediation
    • Escalated the DNS resolution alert through Pager Duty
    • Confirmed the provider's outage by repeatedly calling its hotline and watching its status page
    • Emailed the provider's support, which replied at 08:20 that it was looking into the issue
    • Tested DNS resolution and confirmed that DNS had been restarted
    • Posted a public update on Slack and Twitter announcing the recovery
    Action items
    • Configure a second DNS provider alongside the first
    • Build a public status page for real-time outage information
    • Review and improve internal escalation procedures for third-party outages
    Lessons

    Relying on a single DNS provider made it a single point of failure that brought down every connected service, and the provider's outage also slowed communication with it, which lengthened the downtime. Octez Connect concluded that it needs redundant DNS providers.

    View case
Check your cluster for this

This case involves Authoritative DNS servers and records. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Dependency failure1 case

Component
Authoritative DNS servers and records
Contributing factor
Limited debug accessSingle point of failure
Organisation
Octez Connect
  1. IL-0269 Octez Connect services went down for hours when its only DNS provider failed and the nodes' records stopped resolving
    Octez Connect2025