IL-0310 Hightouch · 2025-10-23databaseaws
An Aurora PostgreSQL failover to an upsized reader reversed twice and backend writes failed in read-only transactions
Hightouch Events uses Amazon Aurora PostgreSQL in us-east-1 as the metadata store for its Postgres-based queues, alongside Kafka and Kubernetes clusters. Aurora separates compute from a shared storage layer replicated across availability zones, and a failover promotes a reader, gives it write access to the storage and points the cluster endpoint at it. After the AWS us-east-1 outage of 20 October 2025 left a large backlog, Hightouch decided to upsize the database on 23 October without a scheduled maintenance window. The plan was to add a temporary reader (instance #3), upsize the existing reader (instance #2) and give it the highest failover priority, fail over to it with an expected downtime under 15s, then upsize the old writer (instance #1) as a reader and remove instance #3. AWS documentation supported the plan, and it had been tested in staging during a load test.
The failovers at 16:39 and 16:43 EDT both reversed. From the database logs Hightouch formed the hypothesis that during the failover window Aurora briefly let both instances process writes, the distributed storage layer rejected the concurrent writes, and both instances crashed: a race between demoting the old writer and accepting writes on the new one. To test this, Hightouch scaled down all services that write to the database, triggered the failover again and watched for storage runtime crashes. Without concurrent writes the failover completed, which strongly reinforced the hypothesis, and the cluster was upsized.
AWS reviewed the findings and confirmed the bug: a signaling problem inside the demotion of the old writer left the writer unchanged after the failover, and nothing in Hightouch's configuration or usage triggered it. AWS said a fix is on its roadmap and, until then, recommends using Aurora failover only as needed and making sure no writes run against the database during a failover.
- Severity
- –
- Scope
- Hightouch Events backend services could not run write queries during two failover attempts until they were restarted, and the planned database upsize failed until a controlled failover with all database writers scaled down
- Users affected
- –
- Region
- us-east-1
- Time to detect
- –
- Time to resolve
- –
Symptoms
At 16:39 EDT on 23 October 2025, Hightouch triggered a failover of its Aurora PostgreSQL cluster in us-east-1 to a newly upsized reader. The AWS Console showed the usual parameter adjustments and instance restarts, but after the page refreshed the original writer was still the primary, and backend services could not run write queries although AWS showed the cluster as healthy.
Backend application logs in some pods showed "DatabaseError: cannot execute UPDATE in a read-only transaction". Database metrics showed a spike in connection count, network traffic and commit throughput on the promoted reader (instance #2) during the failover, while read query traffic from the app did not change, so the extra traffic came from backend services that are meant to reach the writer through the cluster endpoint.
The database logs of both the promoted reader and the original writer showed the same sequence after startup: "LOG: server process (PID 799) was terminated by signal 9: Killed", "DETAIL: Failed process was running: <write query from backend application>", "LOG: terminating any other active server processes", "FATAL: Can't handle storage runtime process crash" and "LOG: database system is shut down".
A second attempt at 16:43 ended the same way, with a brief promotion followed by an immediate reversal. Nothing else had changed: no code updates, no unusual queries and no traffic spikes. The same procedure had passed in a staging environment under load earlier that day, a review of the procedure found no mistake and an online search found no one with the same problem. Restarting the backend services cleared the write errors, but the upgrade had not taken place.
Symptom Failover failureWrites rejected
Root cause
Root cause Software defect
- Trigger
- A planned upsize of the Aurora cluster's instances, done without a maintenance window, after the AWS us-east-1 outage of 20 October had left an event backlog that strained the small Postgres instance. The manual failover to the upsized reader at 16:39 EDT started the failure.
Detection
The AWS Console showed after a refresh that the original writer was still primary, and backend application logs showed write errors on a read-only transaction. Per-instance query monitoring in Datadog and the RDS database logs showed the brief writer advertisement.
Remediation
- Restarted the backend services, which cleared the write errors after each failed failover
- Scaled down every service that writes to the database
- Triggered the failover again with no concurrent writes, which completed, and upsized the cluster in us-east-1
Action items
- Pause writers before any intentional failover, as written into the internal playbooks
- Monitor for unexpected flips of the instance that advertises the writer role
Lessons
Hightouch's takeaways are that a migration can end in the desired state, the starting state or somewhere in between even with services one trusts, so traffic redirection and brief dependency outages should be planned for. The brief writer advertisement could only be found because queries to each instance were monitored in Datadog and the RDS database logs were available. Limiting how much any single component can affect the system helps both uptime and maintenance, and a staging load test, even of the exact procedure, did not reproduce the conditions that triggered the Aurora race condition.
Sources
Related cases
IL-0241A database shard could not elect a new primary, and the Stripe API failed most requests in two periods on one dayStripe · 2019same root cause · shared Database, Database failover manager
IL-0303GitHub's MySQL master failed over to a node with a cold buffer pool during a schema migration and GitHub.com went downGitHub · 2012same root cause · shared Database, Database failover manager
IL-0451Setting indexes INVISIBLE on Aurora MySQL 3.10 spiked CPU and ReadIOPS on all three readers but not on the writerDaangn · 2026same root cause · shared Database
IL-0313Decagon voice requests stalled for exactly 300 seconds behind self-managed PgBouncer and exhausted the SQLAlchemy poolDecagon · 2026same root cause · shared Database
IL-0281PostHog's Logs ClickHouse cluster silently deleted almost all its S3 data, losing US logs older than 3 daysPostHog · 2026same root cause · shared Database
IL-0150A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1Amazon Web Services · 2025same root cause · shared Database
IL-0436A supremum lock left by ROLLBACK TO SAVEPOINT in a long transaction blocked inserts and filled Aurora MySQL connectionsfreee · 2025same root cause · shared Database
IL-0527OpenStreetMap.org made no replication diffs for 16 hours as PostgreSQL logical decoding failed on a memory alloc errorOpenStreetMap Foundation · 2025same root cause · shared Database
IL-0329A routine migration left PGAudit stuck holding Postgres locks, causing intermittent timeouts and a 2-minute outageincident.io · 2025same root cause · shared Database
IL-0347About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuckScaleway · 2024same root cause · shared Database
IL-0373Heroku Postgres Essential-tier databases in the EU region ran out of memory and refused connections, twice in four daysHeroku · 2024same root cause · shared Database
IL-0024Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three timesHugging Face · 2024same root cause · shared Database
IL-0217Honeycomb's main MySQL database locked up under cache-miss reads and ran out of connections, a 68-minute total outageHoneycomb · 2023same root cause · shared Database
IL-0330Requests timed out for two weeks waiting on Go's database/sql connection pool, one for almost 20 secondsincident.io · 2023same root cause · shared Database
IL-0295Pusher's Channels publish API on the mt1 cluster was largely unavailable for 11 minutes while a Redis master failed overPusher · 2022same root cause · shared Database failover manager
IL-0385GitHub MySQL read replicas hit a semaphore deadlock during a schema migration's table rename and kept crash-recoveringGitHub · 2021same root cause · shared Database
IL-0462Some MySQL pods on Shopify's GKE clusters took up to 30 minutes to start, slowing instance replacementShopify · 2021same root cause · shared Database
IL-0196Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the dataPalark · 2020same root cause · shared Database
IL-0486Figma's PostgreSQL 9 database on RDS slowed under high write IOPS and lock contention until the site was unavailableFigma · 2020same root cause · shared Database
IL-0439Database access slowed after each block device and filesystem expansion because the Linux kernel dropped the page cacheCybozu · 2018same root cause · shared Database
IL-0388A Google Cloud live migration took a Discord Redis primary offline, and the failover broke API instances and then guildsDiscord · 2017same root cause · shared Database
IL-0454Baemin billing servers could not open new MariaDB connections at the start of a card-discount eventWoowa Brothers · 2016same root cause · shared Database
IL-0536Amazon SimpleDB in US East returned HTTP 500 to nearly all API calls for two hours after its nodes left the clusterAmazon Web Services · 2014same root cause · shared Database
IL-0485Dropbox went down when an OS upgrade script reinstalled active MySQL master-replica pairs and backup restores ran slowDropbox · 2014same root cause · shared Database
IL-0589Every Kickstarter MySQL replica on Amazon RDS stopped replicating on a duplicate-key error in the backings tableKickstarter · 2013same root cause · shared Database
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0310
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0310/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.