Cases2 cases

1 to 2 of 2

1. IL-0310 An Aurora PostgreSQL failover to an upsized reader reversed twice and backend writes failed in read-only transactions

   Hightouch2025

   Symptoms

   At 16:39 EDT on 23 October 2025, Hightouch triggered a failover of its Aurora PostgreSQL cluster in us-east-1 to a newly upsized reader. The AWS Console showed the usual parameter adjustments and instance restarts, but after the page refreshed the original writer was still the primary, and backend services could not run write queries although AWS showed the cluster as healthy.

   Failover failureWrites rejected

   Trigger

   A planned upsize of the Aurora cluster's instances, done without a maintenance window, after the AWS us-east-1 outage of 20 October had left an event backlog that strained the small Postgres instance. The manual failover to the upsized reader at 16:39 EDT started the failure.

   Root cause

   Software defect

   Component

   DatabaseDatabase failover manager

   Contributing factor

   Unrepresentative staging

   Detection

   The AWS Console showed after a refresh that the original writer was still primary, and backend application logs showed write errors on a read-only transaction. Per-instance query monitoring in Datadog and the RDS database logs showed the brief writer advertisement.

   Remediation

   • Restarted the backend services, which cleared the write errors after each failed failover
   • Scaled down every service that writes to the database
   • Triggered the failover again with no concurrent writes, which completed, and upsized the cluster in us-east-1

   Action items

   • Pause writers before any intentional failover, as written into the internal playbooks
   • Monitor for unexpected flips of the instance that advertises the writer role

   Lessons

   Hightouch's takeaways are that a migration can end in the desired state, the starting state or somewhere in between even with services one trusts, so traffic redirection and brief dependency outages should be planned for. The brief writer advertisement could only be found because queries to each instance were monitored in Datadog and the RDS database logs were available. Limiting how much any single component can affect the system helps both uptime and maintenance, and a staging load test, even of the exact procedure, did not reproduce the conditions that triggered the Aurora race condition.

   View case
2. IL-0241 A database shard could not elect a new primary, and the Stripe API failed most requests in two periods on one day

   Stripe2019

   Symptoms

   On 10 July 2019 the Stripe API failed a substantial majority of requests twice, from 16:35 to 17:02 UTC and from 21:14 to 22:47 UTC. In the first period automated monitoring detected within a minute that a database cluster had failed to elect a new primary.

   High error rateFailover failureWrites rejectedIntermittent timeoutsSustained high node CPU

   Trigger

   The primary node of a database cluster failed at 16:35 UTC while one shard had several stalled nodes. The second period was set off by rolling the election protocol back to the previous minor version at 20:42 UTC.

   Root cause

   Software defect

   Component

   DatabaseDatabase failover manager

   Contributing factor

   Change with a delayed effectMonitoring blind spotExcessive blast radiusHarmful manual actionUnverified assumptionOverlapping failures

   Detection

   Automated monitoring detected the failed election within a minute, and the team was alerted at 16:36 UTC. The second period was announced at 21:14 UTC by automated alerts that some shards in the cluster were unavailable.

   Remediation

   • Restarted all nodes of the database cluster, which forced a successful primary election
   • Updated the production configuration and restarted the database cluster's nodes
   • Ramped traffic back up, putting services for user-initiated API requests first

   Action items

   • Alert when nodes stop reporting their replication lag
   • Alert when a shard enters a state that could trigger the election fault
   • Work with the database maintainers on a fix for the underlying fault
   • Add circuit breaking on failed operations to particular database clusters
   • Use fault isolation to contain the impact of a single failed shard
   • Limit the resources used by clients that keep retrying failed requests
   • Add procedures and tooling that make rapid configuration changes during incident response safer

   View case

By root cause1