IL-0376 Upstash · 2025-02-25databaseaws

A maintenance update set off needless full syncs between Upstash Redis replicas in eu-west-1, slowing the databases

During a maintenance update to the regional Upstash Redis databases in AWS eu-west-1, several databases in the region started a full synchronisation between their primary and backup replicas although none was needed. A full synchronisation throws away all data on the target replica and fills it again from the source replica; it is normally needed only when one replica has lost data integrity, and that was not the case here. The synchronisations drove CPU load very high, and the Upstash system throttled the databases going through them so that the synchronisation got more CPU and finished sooner, which degraded performance on some databases with a replica in the region. Upstash added resources to the cluster, unthrottled the affected databases at 15:06 UTC and enabled more throughput, but latency stayed high until the full synchronisations completed at 21:23 UTC. No data was lost and consistency was kept. Upstash Global databases, Upstash's newer infrastructure and now its default offering, do not have this issue.

Severity: –
Scope: Performance of regional Upstash Redis databases with a replica in AWS eu-west-1: very high CPU load, throttling and high latency; no data or consistency lost
Users affected: –
Region: AWS eu-west-1
Time to detect: –
Time to resolve: –

Symptoms

On 25 February 2025, regional Upstash Redis databases in AWS eu-west-1 slowed down after a maintenance update, with very high CPU load on databases that have a replica in that region. Latency stayed high until about 21:23 UTC, even after the affected databases were unthrottled at 15:06 UTC.

Symptom High latency Sustained high node CPU

Root cause

Root cause Software defect

Component In-memory cache Database replication

Contributing factor Automation amplification

Trigger: A maintenance update to the regional Upstash Redis databases in AWS eu-west-1.

Detection

–

Remediation

Added more resources to the regional eu-west-1 cluster
Unthrottled the affected databases at 15:06 UTC and enabled more throughput

Action items

Deploy a fix that stops regional databases from starting unnecessary full synchronisations
Contact regional database users about migrating to Upstash Global, which does not have this issue

Lessons

–

Sources

https://status.upstash.com/incidents/bczmd5tg3rbj · official-status · archived copy

Related cases

IL-0281 PostHog's Logs ClickHouse cluster silently deleted almost all its S3 data, losing US logs older than 3 daysPostHog · 2026
same root cause · shared Database replication
IL-0527 OpenStreetMap.org made no replication diffs for 16 hours as PostgreSQL logical decoding failed on a memory alloc errorOpenStreetMap Foundation · 2025
same root cause · shared Database replication
IL-0216 Wikimedia on-wiki search was slow or rejected after a datacenter switchover overloaded one Elasticsearch node in codfwWikimedia Foundation · 2023
same root cause · shared In-memory cache
IL-0217 Honeycomb's main MySQL database locked up under cache-miss reads and ran out of connections, a 68-minute total outageHoneycomb · 2023
same root cause · shared In-memory cache
IL-0295 Pusher's Channels publish API on the mt1 cluster was largely unavailable for 11 minutes while a Redis master failed overPusher · 2022
same root cause · shared In-memory cache
IL-0213 Roblox was down for 73 hours because key-value writes on its single Consul cluster slowed from under 300 ms to 2 secondsRoblox · 2021
same root cause · shared In-memory cache
IL-0279 One failed Redis Cluster replica left only about 1 in 21 calls to Grab's Apollo service succeeding for a minuteGrab · 2019
same root cause · shared In-memory cache
IL-0388 A Google Cloud live migration took a Discord Redis primary offline, and the failover broke API instances and then guildsDiscord · 2017
same root cause · shared In-memory cache
IL-0589 Every Kickstarter MySQL replica on Amazon RDS stopped replicating on a duplicate-key error in the backings tableKickstarter · 2013
same root cause · shared Database replication
IL-0303 GitHub's MySQL master failed over to a node with a cold buffer pool during a schema migration and GitHub.com went downGitHub · 2012
same root cause · shared Database replication

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, In-memory cache, Database replication among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0376
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0376/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.