Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0359 Queries to Gandi LiveDNS timed out when two crashed anycast DNS nodes kept announcing all three name server addresses
    Gandi2020
    Symptoms

    On 16 July 2020 between 13:26 and 13:48 UTC, DNS queries to two nodes of Gandi's LiveDNS platform failed with a timeout, and retries against the other name servers of a domain failed as well. The status page reported impact for FR customers.

    DNS resolution failures
    Trigger

    A software bug in Gandi's DNS server software triggered a fault and stopped the server on two LiveDNS nodes.

    Root cause
    Software defect
    Component
    Authoritative DNS servers and recordsBGP routing
    Contributing factor
    Single point of failureSignal buried in noiseOverlapping failures
    Detection

    Monitoring alerts fired on the DNS servers, but they were first read as false positives among the noise from an earlier internal network incident.

    Action items
    • Fix the way a node is disabled when its DNS server is not working correctly
    • Fix the node configuration and add monitoring so that no node announces several clusters
    • Rework monitoring and internal training and organisation
    • Deploy the fix for the bug in the DNS server software
    View case
  2. IL-0278 A weekly update put a Redis Cloud management process into an infinite loop, and most nodes of a GCP cluster failed
    Redis2016
    Symptoms

    On 8 February 2016 at 17:45 UTC, four nodes of the Redis Cloud cluster us-central1-1-1.gce.garantiadata.com on Google Cloud Platform stopped responding, four minutes after a first node had failed. The databases whose endpoints end in us-central1-1-1.gce.garantiadata.com went down.

    Service outageFailover failure
    Trigger

    The weekly upgrade of Redis Cloud's non-critical cluster components reached its first Google Cloud Platform cluster, us-central1-1-1.gce.garantiadata.com, at 17:30 UTC.

    Root cause
    Faulty deployment
    Component
    In-memory cacheDatabase failover managerAuthoritative DNS servers and records
    Contributing factor
    Overlapping failures
    Remediation
    • Shut down and terminated all nodes of the cluster
    • Deleted all of the cluster's endpoints from its DNS zone, removing by hand the entries left by throttled deletes
    • Provisioned new cluster nodes from scratch without the weekly update
    • Reconnected the original nodes' storage to the new nodes so persistence-enabled databases could be recovered
    • Bootstrapped and launched the databases, recovering the ones with persistence from storage
    Action items
    • Suspend all automatic weekly upgrades until a procedure that prevents this kind of outage is in place
    • Revalidate the QA processes for DevOps code deployments
    • Improve the tooling so it handles throttling of DNS requests
    View case
Check your cluster for this

These 2 cases involve Authoritative DNS servers and records and BGP routing. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Software defect1 case

Component
BGP routingAuthoritative DNS servers and records
Contributing factor
Signal buried in noiseOverlapping failuresSingle point of failure
Organisation
Gandi
  1. IL-0359 Queries to Gandi LiveDNS timed out when two crashed anycast DNS nodes kept announcing all three name server addresses
    Gandi2020

Faulty deployment1 case

Component
In-memory cacheAuthoritative DNS servers and recordsDatabase failover manager
Contributing factor
Overlapping failures
Organisation
Redis
  1. IL-0278 A weekly update put a Redis Cloud management process into an infinite loop, and most nodes of a GCP cluster failed
    Redis2016