IL-0359 Gandi · 2020-07-16networking

Queries to Gandi LiveDNS timed out when two crashed anycast DNS nodes kept announcing all three name server addresses

On 16 July 2020 a software bug stopped the DNS server on two of Gandi's LiveDNS nodes. Three problems turned this into an outage. The failsafe that should withdraw a node's anycast BGP route when its DNS server crashes did not work, so the crashed nodes kept attracting queries. A fault in the node configuration made those two nodes announce more anycast addresses than intended, covering all three name servers, a, b and c, so the redundancy between name servers did not help. And an internal network incident that Gandi was already dealing with had flooded monitoring with noise, so the alerts from the DNS servers were taken for false positives. Queries to the two nodes timed out from 13:26 until 13:48 UTC, and the incident was marked resolved at 13:49. Gandi had already tracked down the bug in its DNS server software and was deploying a fix when it posted the postmortem.

Severity: –
Scope: DNS queries for domains on Gandi LiveDNS ({abc}.dns.gandi.net) that reached the two failed nodes timed out
Users affected: –
Region: FR customers, per the status page
Time to detect: –
Time to resolve: 22m

Symptoms

On 16 July 2020 between 13:26 and 13:48 UTC, DNS queries to two nodes of Gandi's LiveDNS platform failed with a timeout, and retries against the other name servers of a domain failed as well. The status page reported impact for FR customers.

LiveDNS nodes are anycast: each node announces its name server addresses in BGP. Domains on LiveDNS use three name servers, ns-x-a.gandi.net, ns-x-b.gandi.net and ns-x-c.gandi.net, and a resolver normally tries the b and then the c server when the a server does not answer. The two broken nodes were announcing all three anycast addresses, 173.246.100.207 (ns-206-a.gandi.net), 213.167.230.65 (ns-64-b.gandi.net) and 217.70.187.111 (ns-110-c.gandi.net), so every query and every retry reached a broken node.

Alerts did fire on the DNS servers, but Gandi was already handling an internal network incident that had filled its monitoring with noise. The team read the LiveDNS alerts as false positives, and the time it took to re-evaluate them made the outage longer.

Symptom DNS resolution failures

Root cause

Root cause Software defect

Component Authoritative DNS servers and records BGP routing

Contributing factor Single point of failure Signal buried in noise Overlapping failures

Trigger: A software bug in Gandi's DNS server software triggered a fault and stopped the server on two LiveDNS nodes.

Detection

Monitoring alerts fired on the DNS servers, but they were first read as false positives among the noise from an earlier internal network incident.

Remediation

–

Action items

Fix the way a node is disabled when its DNS server is not working correctly
Fix the node configuration and add monitoring so that no node announces several clusters
Rework monitoring and internal training and organisation
Deploy the fix for the bug in the DNS server software

Lessons

–

Sources

https://status.gandi.net/incidents/4km3l92vtjvq · official-status · archived copy

Related cases

IL-0226 Azure Front Door edge servers crashed worldwide on incompatible configuration metadata, causing timeouts and DNS errorsMicrosoft Azure · 2025
same root cause · shared Authoritative DNS servers and records
IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1Amazon Web Services · 2025
same root cause · shared Authoritative DNS servers and records
IL-0233 A VPC control plane release broke IP alias routes in Nebius eu-west1, and Managed Kubernetes pods lost connectivityNebius · 2025
same root cause · shared BGP routing
IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuckScaleway · 2024
same root cause · shared BGP routing
IL-0342 DigitalOcean's AMS3, LON1 and SFO2 data centers lost Internet connectivity in repeated waves as edge routers failedDigitalOcean · 2023
same root cause · shared BGP routing
IL-0250 slack.com stopped resolving on some DNSSEC-validating resolvers for about 24 hours after DNSSEC signing was turned onSlack · 2021
same root cause · shared Authoritative DNS servers and records
IL-0222 Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduledGoogle Cloud · 2019
same root cause · shared BGP routing

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Authoritative DNS servers and records, BGP routing among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0359
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0359/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.