IL-0533 Microsoft Azure · 2024-04-23azurenetworking
Names in chinacloudapp.cn and chinacloudsites.cn stopped resolving as DNS caches expired, breaking Azure China services
To meet a regulatory requirement of the Chinese government, Microsoft ran an internal audit to make sure all its domains had the right ownership and were documented properly. In that audit the ownership of two critical system domains of Azure in China, chinacloudapp.cn and chinacloudsites.cn, was misattributed, and both were flagged as candidates for decommission. The next step of the decommissioning process watches active traffic on a flagged domain, but the management tool that provides DNS zone and hosting information did not cover zones hosted within Azure China, so it reported that the zone file did not exist. Domains at end of life often have no zone file, so this notice raised no alert with the operator. The workflow moved on to its next stage and, at 03:45 CST on 23 April 2024, changed the nameservers of the two domains to inactive servers, a final check meant to find hidden users of a domain.
As DNS caches across the Internet timed out, resolvers fetched the two domains again, received the inactive nameservers and failed to resolve names in them, and many Azure China services, the portal and APIs failed. Internal monitors raised an incident at 04:37 CST, engineers acknowledged it at 04:39 and at 04:57 matched the resolution failures to the nameserver change. The previous nameservers were restored at 04:59 and the reversion was complete at 05:13, when services began to recover. Because resolvers had cached the bad answers, recovery took until 06:00 CST, when traffic for the services and the affected DNS zones was back to the level before the incident.
- Severity
- –
- Scope
- Name resolution in chinacloudapp.cn and chinacloudsites.cn, and through it connections to many Azure services in China, the Azure Portal (portal.azure.cn) and APIs, and Azure Monitor, Log Analytics and Microsoft Sentinel in China East 2 and China East 3
- Users affected
- Customers of the Azure China regions
- Region
- Azure China regions
- Time to detect
- 52m
- Time to resolve
- 2h 15m
Symptoms
From 03:45 CST on 23 April 2024, FQDNs in two system domains of Azure in China, chinacloudapp.cn and chinacloudsites.cn, gradually stopped resolving, and customers in the Azure China regions had trouble connecting to many Azure services and managing resources. Internal monitors alerted engineers to the degradation at 04:37 CST.
The impact was not immediate. Resolvers across the Internet kept the old records until their TTLs ran out, and only when the resolvers refreshed the two domains did the resolvers receive the new, inactive nameservers and fail to resolve names. Customers then had trouble connecting to Cosmos DB, Azure Virtual Desktop, Azure Databricks, Backup, Site Recovery, Azure IoT Hub, Service Bus, Logic Apps, Data Factory, Azure Kubernetes Services, Azure Policy, Azure AI Speech, Azure Machine Learning, API Management, Azure Container Registry and Azure Data Explorer, and viewing or managing resources through the Azure Portal (portal.azure.cn) or APIs failed.
Azure Monitor, Log Analytics and Microsoft Sentinel in China East 2 and China East 3 had intermittent data latency and failed to query and retrieve data, which could stop alerts from firing, and create, update, retrieve and delete operations in Log Analytics could fail. Only these Microsoft-owned domains were hit; the Azure DNS platform and other zones hosted on Azure kept working.
Symptom DNS resolution failuresHigh error rateMonitoring outage
Root cause
Root cause Configuration change
- Trigger
- At 03:45 CST a configuration change made through a domain registrar, as a step of a domain decommissioning workflow, updated the nameservers of chinacloudapp.cn and chinacloudsites.cn to a set of inactive servers.
Detection
Internal monitors alerted engineers to the degradation in the Azure China Cloud at 04:37 CST and created an incident, which the engineering team acknowledged at 04:39 CST.
Remediation
- Reverted the two domains to their previous working nameservers
Action items
- Suspended further runs of the domain lifecycle process until the management tool covers Azure in China
- Make the validation process of domain lifecycle management include signals from all cloud regions
- Add validations that get nameserver information by resolving the zone directly over the Internet
Lessons
–
Sources
- https://azure.status.microsoft/en-us/status/history/?trackingId=V_BH-7DZ · official-status
Related cases
IL-0512xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failedMIT Open Learning · 2026 · sev1same root cause · shared Authoritative DNS servers and records
IL-0502The JupyterHub at temple.2i2c.cloud stopped resolving in DNS after a wildcard record was added for a new cluster2i2c · 2025same root cause · shared Authoritative DNS servers and records
IL-0254Square's internal DNS servers failed and Square products, services and payments were unavailable for about 15 hoursSquare · 2023same root cause · shared Authoritative DNS servers and records
IL-0549Media, attachments, Git LFS and build artifacts failed for Atlassian in us-west-1 when an old network stack was removedAtlassian · 2023same root cause · shared Authoritative DNS servers and records
IL-0363static.crates.io did not resolve for 10-15 minutes during a Terraform deployment, so Rust builds could not fetch cratesRust Project · 2023same root cause · shared Authoritative DNS servers and records
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0533
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0533/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.