Azure Database for MySQL Single Server and MariaDB in West US rejected every new connection for a batch of servers

Symptoms

From 04:26 UTC on 19 April 2024, every new connection to a subset of Azure Database for MySQL - Single Server and Azure Database for MariaDB servers in West US failed, and retries did not help. The first report came at 04:46 UTC from Azure Databricks, which told the MySQL team that multiple databases were unreachable.

Azure Databricks depends on the affected database service, so Databricks customers in West US, West US 2 and South Central US saw workspace login and authentication requests fail or time out. Cluster CRUD requests and jobs that needed a cluster to start, resize or terminate did not run, jobs submitted through APIs or schedulers did not run, UI and Databricks SQL queries timed out, Databricks SQL Serverless Warehouses failed to launch and UC APIs could not be reached. The errors cited "Authentication is temporarily unavailable" or "TEMPORARILY_UNAVAILABLE".

At 05:14 UTC the MySQL on-call engineers found that the databases themselves were healthy but no connections were reaching them. The networking team joined at 05:44 UTC. Service telemetry showed nothing, because the gateway ring that received the connections was not fully configured and had no telemetry or monitoring yet. At 06:46 UTC the failures were linked to the move of traffic to a new gateway ring.

Symptom Service outageIntermittent timeouts

Root cause

Trigger

At 04:26 UTC, during planned maintenance in West US, the DNS update that moves new incoming connections for a batch of MySQL and MariaDB servers to a new gateway ring was made, and a procedural error pointed it at a newly built gateway ring whose configuration was incomplete.

Detection

Service telemetry did not detect the failure because the new gateway ring had no telemetry or monitoring configured yet. At 04:46 UTC Azure Databricks reported to the MySQL team that multiple databases were unreachable.

Remediation

• Rolled back the DNS change and moved all connections back to the previous working gateway ring after validation

Action items

• Add checks, testing and sign-off procedures before switching production gateway rings
• Add Azure Databricks alerts that detect multiple databases becoming unavailable within a short time

Lessons

–

Sources

• https://azure.status.microsoft/en-us/status/history/?trackingId=QS3K-BPZ · official-status

References

• https://learn.microsoft.com/azure/databricks/administration-guide/disaster-recovery
• https://learn.microsoft.com/azure/postgresql/flexible-server/overview

Related cases

• IL-0577 Ficilcom's production Neon PostgreSQL database was reset by prisma migrate reset meant for the local databaseFicilcom · 2026

  same root cause · shared Database

• IL-0576 Every dataset in CareNet's production BigQuery data warehouse was deleted, and UNDROP SCHEMA could not restore themCareNet · 2025

  same root cause · shared Database

• IL-0242 Both PostgreSQL servers behind matrix.org were lost during a disk expansion, and the homeserver was down for 24 hoursMatrix.org Foundation · 2025

  same root cause · shared Database

• IL-0348 Scaleway's FR-PAR region lost public internet connectivity when one 100 Gbps backbone link was saturatedScaleway · 2025

  same root cause · shared Authoritative DNS servers and records

• IL-0582 Jobs piled up in Kanmu's PostgreSQL-based qg job queue on Aurora as dequeue slowed and dead tuples kept growingKanmu · 2024

  same root cause · shared Database

• IL-0556 A transaction left open during database maintenance stopped uploads and then downloads on Storj's US1 satelliteStorj · 2024

  same root cause · shared Database

• IL-0565 Cookie Run: Kingdom's CockroachDB cluster failed when 16 of 24 nodes lost their filesystems, causing a 36-hour outageDevsisters · 2022

  same root cause · shared Database

• IL-0207 A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all trafficGitLab · 2022

  same root cause · shared Database

• IL-0221 A backbone maintenance command disconnected all Facebook data centers, and its DNS servers withdrew their BGP routesMeta · 2021

  same root cause · shared Authoritative DNS servers and records

• IL-0332 A test run against the production database truncated every table and travis-ci.com was down for about 5.5 hoursTravis CI · 2018

  same root cause · shared Database

• IL-0431 During Mackerel's move to AWS, PostgreSQL failed over to a replica that had stopped replicating, losing an hour of dataHatena · 2017

  same root cause · shared Database

• IL-0506 UC Berkeley DataHub logins failed and students got empty home directories after the JupyterHub database was overwrittenUC Berkeley · 2017

  same root cause · shared Database

• IL-0218 GitLab.com's primary PostgreSQL data was wiped by mistake and, with no working backup, the site was down about 18 hoursGitLab · 2017

  same root cause · shared Database

• IL-0573 A developer's mistake dropped nine main tables of the Baemin billing MariaDB database and payments briefly stoppedWoowa Brothers · 2016

  same root cause · shared Database

• IL-0319 Dropping databases on the MySQL replica mysql2 also dropped them on mysql1, as the pair actually ran master-masterOregon State University Open Source Lab · 2015

  same root cause · shared Database

• IL-0337 EBS volumes in one US East Availability Zone got stuck after a network change cut EBS nodes off from their replicasAmazon Web Services · 2011

  same root cause · shared Database

Check your cluster for this

For AI tools and scripts

• This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0531
• The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0531/similar
• MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.