Incident Library

IL-0367 Microsoft Azure · 2024-07-18azuredatabase

Azure Storage in Central US rejected VM disk I/O, taking down VMs and the SQL Database and Cosmos DB services on them

Azure VMs with persistent disks use disks backed by Azure Storage. As a defence-in-depth measure, Storage scale units accept disk reads and writes only from network address ranges known to belong to the physical hosts that run VMs, and an allow list of those ranges is published to every Storage scale unit in a region as hosts are added and removed, at least once a day in large regions. The list is built from one file per datacenter on a set of infrastructure file servers. On 18 July 2024 some of those file servers could not be reached from the generating server after recent network configuration changes; the workflow did not notice the missing files and published an allow list with incomplete VM host address ranges. The allow list deployment went through the region in batches within about an hour, did not check for drops in VM availability and did not follow Safe Deployment Practices (SDP) such as Availability Zone sequencing, so Storage servers across the region rejected disk requests from the affected VM hosts.

VMs in many control-plane and data-plane clusters failed, and the services built on them failed with them, including Azure SQL Database and Managed Instance, Cosmos DB, the Database for MySQL and PostgreSQL flexible servers, Azure Resource Manager, Azure DevOps, Privileged Identity Management and Azure NetApp Files, and through them parts of Microsoft 365, Dynamics 365 and Microsoft Entra. Once the incomplete allow list was identified as the trigger, Microsoft halted the allow list update workflow worldwide and updated the configuration of every Storage scale unit in Central US, which restored storage availability at 02:55 UTC on 19 July.

Database recovery took most of the next day. For SQL, Microsoft started failovers for databases with automatic failover policies within an hour, promoting the geo-secondary and pointing the failover group (FOG) endpoint to the new primary. Less than 0.5% of those failovers did not complete, because failover workflows were terminated or throttled under high demand on the service manager component, and engineers forced those failovers to completion by hand. After storage came back 98% of SQL databases recovered, while about 2% needed gateway node restarts because the metadata on the gateway nodes was out of sync with replica placement; all SQL databases were verified as recovered at 22:00 UTC on 19 July. Cosmos DB failed over its control plane and the active-passive accounts whose write region was Central US, finishing 95% of account failovers by 02:29 UTC; the other 5% needed customer-started region switches cancelled or locks on service metadata removed. Single-region Cosmos DB accounts were brought back through partition repairs, and failback to Central US had to be redone for some MongoDB API accounts and for accounts with private endpoints.

Severity
–
Scope
VM availability in Central US and the services on those VMs, including SQL Database, SQL Managed Instance, Cosmos DB, Database for MariaDB, Database for MySQL and PostgreSQL flexible servers and Azure Resource Manager; new Azure NetApp Files volumes could not be created in any region
Users affected
–
Region
Central US
Time to detect
26m
Time to resolve
24h 20m

Symptoms

From 21:40 UTC on 18 July 2024, virtual machines in the Azure Central US region lost access to their disks, and the services running on those VMs failed. Azure SQL Database and SQL Managed Instance showed unhealthy nodes and login failures, Cosmos DB requests failed, and service management operations failed across many Azure services.

Storage scale units rejected every disk request from the VM hosts whose address ranges were missing from the storage allow list. Service monitoring saw VM availability drop at 22:06 UTC, and VM availability reached its lowest point at 22:45 UTC, when the deployment of the incomplete allow list finished. Storage scale units that host Premium v2 and Ultra Disk were not affected.

The Cosmos DB request success rate in Central US fell to 82% at its lowest point, with about 50% of the VMs running Cosmos DB in the region down; the infrastructure failed step by step over 68 minutes across several availability zones. Some applications saw errors even though their Cosmos DB accounts could serve traffic in other regions, which the first investigation traced to client-side timeouts. For Azure SQL, less than 0.5% of the databases that were failed over did not finish the failover, and after storage recovered about 2% of databases stayed unavailable because gateway nodes routed connections using metadata that no longer matched where the replicas were.

Because the impact was widespread and the first visible symptom looked like a drop in VM disk traffic to Storage scale units, engineers needed time to rule out other causes before linking the failures to the incomplete storage allow list, which was confirmed as the cause at 00:45 UTC.

Symptom Storage disconnected or stale mountsService outageHigh error rateFailover failureStale endpoint routing

Root cause

Root cause Configuration change

Component Cloud block storageDatabaseDatabase failover manager
Contributing factor Unsafe failure behaviourNo staged rolloutExcessive blast radiusMonitoring blind spot
Trigger
Routine changes to the VM host fleet led to a new allow list of VM host address ranges being generated and published to all Storage scale units in Central US. Recent network configuration changes had made some of the infrastructure file servers that hold the source data unreachable from the server generating the list, so the published list was incomplete.

Detection

Service monitoring detected the drop in VM availability at 22:06 UTC while the allow list update was being published. SQL monitoring detected unhealthy nodes and login failures within one minute of SQL becoming unavailable, and monitoring flagged the availability drop for Cosmos DB, SQL DB and SQL DB Managed Instance at 22:14 UTC.

Remediation

Action items

Lessons

–

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Cloud block storage, Database, Database failover manager among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.