Cases18 cases

1 to 5 of 18

1. IL-0569 After an MHA master failover, one of three MySQL replicas failed with a malformed binlog error and could not catch up

   DeNA2026

   Symptoms

   On the event day of a global game, with close to 1000 rps per MySQL instance even after sharding and replication, a master instance stopped because of a host failure. MHA failed over automatically as usual and the service kept running, but one of the three replicas hit a replication error and could no longer follow the latest data.

   Replication lag

   Trigger

   Under heavy load on a game event day, a MySQL master instance stopped because of a host failure, and MHA started an automatic failover.

   Root cause

   Breaking change in an upgrade

   Component

   Database failover managerDatabase replicationDatabase

   Contributing factor

   Change with a delayed effect

   Remediation

   • Removed the replica that could not apply data from service

   Lessons

   MHA had not followed a mysqlbinlog change in MySQL 5.7.9, so under specific conditions bringing replicas up to date during failover failed. Running MySQL at large scale means meeting rare events like this, which only show when a master stops suddenly under load.

   View case
2. IL-0310 An Aurora PostgreSQL failover to an upsized reader reversed twice and backend writes failed in read-only transactions

   Hightouch2025

   Symptoms

   At 16:39 EDT on 23 October 2025, Hightouch triggered a failover of its Aurora PostgreSQL cluster in us-east-1 to a newly upsized reader. The AWS Console showed the usual parameter adjustments and instance restarts, but after the page refreshed the original writer was still the primary, and backend services could not run write queries although AWS showed the cluster as healthy.

   Failover failureWrites rejected

   Trigger

   A planned upsize of the Aurora cluster's instances, done without a maintenance window, after the AWS us-east-1 outage of 20 October had left an event backlog that strained the small Postgres instance. The manual failover to the upsized reader at 16:39 EDT started the failure.

   Root cause

   Software defect

   Component

   DatabaseDatabase failover manager

   Contributing factor

   Unrepresentative staging

   Detection

   The AWS Console showed after a refresh that the original writer was still primary, and backend application logs showed write errors on a read-only transaction. Per-instance query monitoring in Datadog and the RDS database logs showed the brief writer advertisement.

   Remediation

   • Restarted the backend services, which cleared the write errors after each failed failover
   • Scaled down every service that writes to the database
   • Triggered the failover again with no concurrent writes, which completed, and upsized the cluster in us-east-1

   Action items

   • Pause writers before any intentional failover, as written into the internal playbooks
   • Monitor for unexpected flips of the instance that advertises the writer role

   Lessons

   Hightouch's takeaways are that a migration can end in the desired state, the starting state or somewhere in between even with services one trusts, so traffic redirection and brief dependency outages should be planned for. The brief writer advertisement could only be found because queries to each instance were monitored in Datadog and the RDS database logs were available. Limiting how much any single component can affect the system helps both uptime and maintenance, and a staging load test, even of the exact procedure, did not reproduce the conditions that triggered the Aurora race condition.

   View case
3. IL-0306 A Neon pageserver's EC2 instance failed without warning and Postgres databases of 0.4% of us-east-1 projects stalled

   Neon2024

   Symptoms

   At 17:28 on 2024-08-14 the EC2 instance pageserver-7.us-east-1.aws.neon.tech stopped responding. Neon customer databases attached to that pageserver stalled whenever they needed to read data that was not already in their Postgres buffer cache.

   Storage disconnected or stale mountsHigh error rate

   Trigger

   The EC2 instance hosting pageserver-7 in us-east-1 failed without the advance warning AWS usually gives before a hardware failure.

   Root cause

   Hardware failure

   Component

   DatabaseDatabase failover manager

   Contributing factor

   Partial rollout

   Detection

   Several alerts fired at 17:38, in Slack and as automatic pages to the on-call engineers: the pageserver stopped answering metric scrapes, the control plane API error rate rose, and user operations such as starting endpoints and creating timelines got stuck.

   Remediation

   • Migrated the 17037 projects on pageserver-7 to other pageservers in the region with a semi-manual script
   • Brought back by hand about 20 projects whose in-flight operations blocked the migration

   Action items

   • Move all paying customers onto the storage controller sooner than planned
   • Finish moving the existing pageserver infrastructure under the storage controller

   Lessons

   Neon's first control plane design, in which operations always run to completion, cannot drive a fast response to node failures at scale, because operations that depend on physical machines may never complete. A human decision in the loop and alerts with several minutes of delay made the response too slow; the storage controller, with its own heartbeats and automatic rescheduling, moves work off a failed pageserver in seconds.

   View case
4. IL-0367 Azure Storage in Central US rejected VM disk I/O, taking down VMs and the SQL Database and Cosmos DB services on them

   Microsoft Azure2024

   Symptoms

   From 21:40 UTC on 18 July 2024, virtual machines in the Azure Central US region lost access to their disks, and the services running on those VMs failed. Azure SQL Database and SQL Managed Instance showed unhealthy nodes and login failures, Cosmos DB requests failed, and service management operations failed across many Azure services.

   Storage disconnected or stale mountsService outageHigh error rateFailover failureStale endpoint routing

   Trigger

   Routine changes to the VM host fleet led to a new allow list of VM host address ranges being generated and published to all Storage scale units in Central US. Recent network configuration changes had made some of the infrastructure file servers that hold the source data unreachable from the server generating the list, so the published list was incomplete.

   Root cause

   Configuration change

   Component

   Cloud block storageDatabaseDatabase failover manager

   Contributing factor

   Unsafe failure behaviourNo staged rolloutExcessive blast radiusMonitoring blind spot

   Detection

   Service monitoring detected the drop in VM availability at 22:06 UTC while the allow list update was being published. SQL monitoring detected unhealthy nodes and login failures within one minute of SQL becoming unavailable, and monitoring flagged the availability drop for Cosmos DB, SQL DB and SQL DB Managed Instance at 22:14 UTC.

   Remediation

   • Halted the allow list update workflow worldwide
   • Paused all deployments in the Central US region
   • Paused compute buildout deployments in all regions
   • Updated the configuration of all Storage scale units in Central US, restoring storage availability by 02:55 UTC
   • Started geo-failovers for SQL databases with automatic failover policies
   • Restarted SQL gateway nodes to refresh their caches and route connections to the right nodes
   • Forced completion of SQL failovers that had not completed
   • Failed over the Cosmos DB control plane and active-passive accounts with a write region in Central US
   • Cancelled customer-triggered graceful switch region operations that blocked Cosmos DB failovers
   • Removed locks on Cosmos DB service metadata to force-complete stuck failovers
   • Repaired impacted Cosmos DB partitions for single-region accounts
   • Repaired backend nodes and failed Cosmos DB accounts back to Central US, redoing failback where it failed

   Action items

   • Make the allow list generation workflow detect incomplete source information and halt
   • Alert on storage requests rejected by allow list checks
   • Deploy allow lists one Availability Zone and storage type at a time, over 24 hours
   • Add VM health checks and an automatic stop to the allow list deployment workflow
   • Reevaluate the policy for Microsoft-managed failover of SQL failover groups
   • Fix the Cosmos DB failback workflow that broke some MongoDB drivers for MongoDB API accounts
   • Fix Cosmos DB failback for accounts with private endpoints so it does not need redoing
   • Make the storage data-plane firewall reject invalid allow lists and keep the last-known-good state
   • Improve Azure NetApp Files monitors so impacting events are detected and classified in time
   • Monitor more Azure NetApp Files service metrics to correlate similar events faster
   • Make the SQL Service Fabric cluster location change notifications reliable under load
   • Make SQL geo-failover workflows more robust so they complete
   • Remove the issues that delayed 5% of Cosmos DB failovers
   • Migrate Azure DevOps customer metadata to the right geography
   • Decouple Azure NetApp Files regional reads and writes to keep this class of issue to one region
   • Evaluate caching to reduce Azure NetApp Files reliance on stored utilization data
   • Add automatic per-partition failover for Cosmos DB multi-region active-passive accounts
   • Build the Resilient Ephemeral OS disk improvement so VMs survive storage incidents better

   View case
5. IL-0291 GitLab.com's redis-sidekiq Sentinel could not elect a primary after a GCP host error, so Sidekiq was down for 30 minutes

   GitLab2023

   Symptoms

   GitLab.com showed widespread, erratic behaviour and more 500 errors for about 30 minutes while its redis-sidekiq Redis was down. Alerts in the #production channel began with Redis client errors and spread to error SLO violations on several services, such as the Webservice and API load balancer and Puma.

   Failover failureHigh error rate

   Trigger

   The VM of the primary node, redis-sidekiq-03-db-gprd, failed on a GCP host error and was reset and moved to another node.

   Root cause

   Cloud provider outage

   Component

   In-memory cacheDatabase failover manager

   Contributing factor

   Single point of failureMonitoring blind spot

   Detection

   Several alerts in the #production channel, first Redis client errors and then error SLO violations on several services, such as the Webservice and API load balancer and Puma.

   Remediation

   • Restarted the failed Redis node so Sentinel had enough nodes to elect a primary

   Action items

   • Alert when the Sentinel service is down
   • Make merge requests robust to Redis Sidekiq outages

   Lessons

   An alert on a down Sentinel node would have kept enough nodes up to form a quorum when failover was needed, and could have avoided the incident entirely. Removing a noisy alert earlier may have let the down node go unnoticed.

   View case

By root cause9