Incident Library
Applied filters ComponentAuthoritative DNS servers and recordsFactorMonitoring blind spotOrganisationMicrosoft AzureClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0531 Azure Database for MySQL Single Server and MariaDB in West US rejected every new connection for a batch of servers
    Microsoft Azure2024
    Symptoms

    From 04:26 UTC on 19 April 2024, every new connection to a subset of Azure Database for MySQL - Single Server and Azure Database for MariaDB servers in West US failed, and retries did not help. The first report came at 04:46 UTC from Azure Databricks, which told the MySQL team that multiple databases were unreachable.

    Service outageIntermittent timeouts
    Trigger

    At 04:26 UTC, during planned maintenance in West US, the DNS update that moves new incoming connections for a batch of MySQL and MariaDB servers to a new gateway ring was made, and a procedural error pointed it at a newly built gateway ring whose configuration was incomplete.

    Root cause
    Operator error
    Component
    DatabaseAuthoritative DNS servers and records
    Contributing factor
    Monitoring blind spot
    Detection

    Service telemetry did not detect the failure because the new gateway ring had no telemetry or monitoring configured yet. At 04:46 UTC Azure Databricks reported to the MySQL team that multiple databases were unreachable.

    Remediation
    • Rolled back the DNS change and moved all connections back to the previous working gateway ring after validation
    Action items
    • Add checks, testing and sign-off procedures before switching production gateway rings
    • Add Azure Databricks alerts that detect multiple databases becoming unavailable within a short time
    View case
Check your cluster for this

This case involves Database and Authoritative DNS servers and records. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Operator error1 case

Component
DatabaseAuthoritative DNS servers and records
Contributing factor
Monitoring blind spot
Organisation
Microsoft Azure
  1. IL-0531 Azure Database for MySQL Single Server and MariaDB in West US rejected every new connection for a batch of servers
    Microsoft Azure2024