Cases4 cases

1 to 4 of 4

1. IL-0310 An Aurora PostgreSQL failover to an upsized reader reversed twice and backend writes failed in read-only transactions

   Hightouch2025

   Symptoms

   At 16:39 EDT on 23 October 2025, Hightouch triggered a failover of its Aurora PostgreSQL cluster in us-east-1 to a newly upsized reader. The AWS Console showed the usual parameter adjustments and instance restarts, but after the page refreshed the original writer was still the primary, and backend services could not run write queries although AWS showed the cluster as healthy.

   Failover failureWrites rejected

   Trigger

   A planned upsize of the Aurora cluster's instances, done without a maintenance window, after the AWS us-east-1 outage of 20 October had left an event backlog that strained the small Postgres instance. The manual failover to the upsized reader at 16:39 EDT started the failure.

   Root cause

   Software defect

   Component

   DatabaseDatabase failover manager

   Contributing factor

   Unrepresentative staging

   Detection

   The AWS Console showed after a refresh that the original writer was still primary, and backend application logs showed write errors on a read-only transaction. Per-instance query monitoring in Datadog and the RDS database logs showed the brief writer advertisement.

   Remediation

   • Restarted the backend services, which cleared the write errors after each failed failover
   • Scaled down every service that writes to the database
   • Triggered the failover again with no concurrent writes, which completed, and upsized the cluster in us-east-1

   Action items

   • Pause writers before any intentional failover, as written into the internal playbooks
   • Monitor for unexpected flips of the instance that advertises the writer role

   Lessons

   Hightouch's takeaways are that a migration can end in the desired state, the starting state or somewhere in between even with services one trusts, so traffic redirection and brief dependency outages should be planned for. The brief writer advertisement could only be found because queries to each instance were monitored in Datadog and the RDS database logs were available. Limiting how much any single component can affect the system helps both uptime and maintenance, and a staging load test, even of the exact procedure, did not reproduce the conditions that triggered the Aurora race condition.

   View case
2. IL-0240 Cloudflare API requests failed and the dashboard slowed for hours after the authentication database lost its replicas

   Cloudflare2020

   Symptoms

   On 2 November 2020 the Cloudflare API and dashboard were degraded for six hours and 33 minutes. The success rate of API queries dropped at times to 75%, and the dashboard was up to 80 times slower than normal, while Cloudflare's edge network kept working.

   High error rateHigh latencyDatabase overloadWrites rejectedPacket loss

   Trigger

   At 14:43 UTC a network switch in a control plane data center partly failed: it kept LACP and BGP running but stopped forwarding some packets, and it recovered on its own six minutes later.

   Root cause

   DNS or network fault

   Component

   NIC, cable and switch port linkCoordination and key-value storeDatabase failover managerDatabaseDatabase replication

   Contributing factor

   Automation amplificationNo headroomHarmful manual action

   Detection

   At 14:43 UTC alerts fired because the misbehaving switch was unreachable to pings. Engineers then saw that the primary authentication database was overwhelmed.

   Remediation

   • Shed load from the primary authentication database by slowing SSL certificate pushes to the edge, emails and other features
   • Moved API traffic that could use read replicas to the secondary data center by hand
   • Sent authentication calls fully back to the primary data center when the dashboard got worse
   • Rebuilt the first database replica, which put itself back into service at 21:20 UTC and returned performance to normal
   • Ramped the turned-down services back up so asynchronous processing could catch up

   Action items

   • Remove the cluster management limitation that requires rebuilding all replicas after a primary is promoted
   • Redesign the user sessions system so dashboard traffic can be steered between data centers
   • Change the auto-remediation configuration so a database replica is not promoted so quickly
   • Tune each system involved to be more resilient to failures of its dependencies

   Lessons

   Every system in the chain had redundancy and none failed completely; each only degraded, which made the chain of events much harder to model and foresee. Automating replica promotion so that it could trigger on a minute's notice had been a point of pride, but for at least one database a fast promotion can do more harm than the failure it answers. Most general-purpose cluster managers use Raft or Paxos rather than Byzantine fault tolerant protocols, accepting a rare failure mode in exchange for a protocol that is simpler to implement correctly and debug.

   View case
3. IL-0241 A database shard could not elect a new primary, and the Stripe API failed most requests in two periods on one day

   Stripe2019

   Symptoms

   On 10 July 2019 the Stripe API failed a substantial majority of requests twice, from 16:35 to 17:02 UTC and from 21:14 to 22:47 UTC. In the first period automated monitoring detected within a minute that a database cluster had failed to elect a new primary.

   High error rateFailover failureWrites rejectedIntermittent timeoutsSustained high node CPU

   Trigger

   The primary node of a database cluster failed at 16:35 UTC while one shard had several stalled nodes. The second period was set off by rolling the election protocol back to the previous minor version at 20:42 UTC.

   Root cause

   Software defect

   Component

   DatabaseDatabase failover manager

   Contributing factor

   Change with a delayed effectMonitoring blind spotExcessive blast radiusHarmful manual actionUnverified assumptionOverlapping failures

   Detection

   Automated monitoring detected the failed election within a minute, and the team was alerted at 16:36 UTC. The second period was announced at 21:14 UTC by automated alerts that some shards in the cluster were unavailable.

   Remediation

   • Restarted all nodes of the database cluster, which forced a successful primary election
   • Updated the production configuration and restarted the database cluster's nodes
   • Ramped traffic back up, putting services for user-initiated API requests first

   Action items

   • Alert when nodes stop reporting their replication lag
   • Alert when a shard enters a state that could trigger the election fault
   • Work with the database maintainers on a fix for the underlying fault
   • Add circuit breaking on failed operations to particular database clusters
   • Use fault isolation to contain the impact of a single failed shard
   • Limit the resources used by clients that keep retrying failed requests
   • Add procedures and tooling that make rapid configuration changes during incident response safer

   View case
4. IL-0320 Postmark's primary MySQL disk filled but MySQL kept running, so MySQL-MMM never failed over and sending was delayed

   Postmark2011

   Symptoms

   At about 8:15am EDT Postmark's primary MySQL server ran out of disk space, and email sending, the API and the web interface were delayed. Disk space alerts fired, but the space had grown much faster than usual and MySQL was already struggling when the alerts were noticed. MySQL kept running while it failed to write some data to disk, so MySQL-MMM, which normally detects problems and fails over to a healthy server, never triggered a failover; the team spent about 30 minutes trying to free logs and disk space before failing over by hand.

   Node disk fullWrites rejectedFailover failure

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase failover manager

   Contributing factor

   –

   Detection

   Per-server monitoring raised disk space alerts, but the disk filled so quickly that MySQL was already struggling by the time the alerts were noticed.

   Remediation

   • Tried to free up logs and disk space on the primary server
   • Failed over manually to the second MySQL server
   • Recovered the jobs that had failed
   • Sent the emails that each server had queued locally while the database was unavailable

   Action items

   • Upgrade the MySQL cluster with a focus on availability and performance
   • Make the alerts more sensitive to catch problems sooner

   Lessons

   Looking back, Postmark said it should have failed over to the second server right away instead of first making sure replication was healthy.

   View case

By root cause3