Cases11 cases

1 to 5 of 11

1. IL-0368 PlanetScale's control plane went down and some database branches failed queries during the AWS us-east-1 outage

   PlanetScale2025

   Symptoms

   At 7:13 UTC on 20 October 2025, PlanetScale's continuous tests in production started failing because they could not create new database branches. The service that creates, resizes and configures database branches was unavailable, a near-total control plane outage, while existing database branches kept their capacity and connectivity.

   Service outageScale-up failureNetwork unreachableHigh error rateReplication lagDNS resolution failures

   Trigger

   The incident started with a DNS misconfiguration at one of PlanetScale's service providers. An Amazon DynamoDB outage impacted AWS STS, and several hours of capacity constraints and network instability in AWS us-east-1 followed.

   Root cause

   Cloud provider outage

   Component

   DatabaseDatabase replicationDatabase proxy and connection poolerObject storage

   Contributing factor

   –

   Detection

   PlanetScale's continuous testing in production started failing and alerted engineers at 7:13 UTC. At 10:05 UTC an alert fired for a Kubernetes operator in a large single-tenant customer installation exhausting its resources.

   Remediation

   • Disallowed new databases in us-east-1 and made us-east-2 the default region for new databases
   • Delayed new backups and cancelled pending backups that were waiting for an EC2 instance
   • Advised PlanetScale Managed customers with vtgate autoscaling to shed load
   • Paused draining and terminating EC2 instances older than 30 days
   • Kept vacant EC2 instances for reuse instead of terminating them
   • Bin-packed vtgate processes more tightly for customers with autoscaling
   • Sent manual reparent requests to move primaries to healthier availability zones
   • Restarted the edge load balancer and vtgate processes that did not recover after the partitions healed

   Action items

   • Understand and become resilient to the failure modes of SaaS dependencies for CI/CD, SSO, web application hosting and incident communication
   • Investigate more ambitious ways to reduce runtime dependence on internal and AWS services
   • Look into using all six us-east-1 availability zones to tolerate zonal outages and partitions

   Lessons

   Keeping the control plane separate from the data plane meant the control plane outage did not affect customer databases, and redundancy, automated failover and zonal traffic routing limited the effect of the network partitions. Network partitions are hard to reason about, test and tolerate: with three availability zones a system survives losing one zone only if the other two can still reach each other reliably.

   View case
2. IL-0242 Both PostgreSQL servers behind matrix.org were lost during a disk expansion, and the homeserver was down for 24 hours

   Matrix.org Foundation2025

   Symptoms

   On 2 September 2025 the matrix.org homeserver went down at 17:45 UTC and stayed down until 18:00 UTC the next day. Both machines of its PostgreSQL cluster were out of action: the original primary had lost its RAID array during a disk expansion, and the data directory of the new primary had been deleted by mistake.

   Service outageData loss

   Trigger

   During maintenance to add drives, an existing drive disappeared from the primary database server's RAID 10 array at 11:17 UTC, which led to a failover and a rebuild of that server. The outage itself began when the data directory of the new primary on db-01 was deleted while clearing a failed restore on db-02.

   Root cause

   Operator error

   Component

   DatabaseDatabase backups and snapshotsDatabase replication

   Contributing factor

   Overlapping failuresHarmful manual actionUnpatched known bugNo safeguard on destructive change

   Detection

   Monitoring fired when the drive disappeared from db-02's RAID array at 11:17 UTC, and Mythic Beasts confirmed the issue. The team realised it had deleted db-01's data directory, and the backend team was then paged, an incident was opened and an emergency was declared.

   Remediation

   • Created a fresh RAID 10 array on db-02
   • Unblocked WAL archiving from the new primary on db-01 to S3
   • Started restoring db-02 from the S3 backups with wal-g
   • Shut down Synapse and the damaged primary database
   • Remounted the data partition of db-01 read-only
   • Copied the WALs missing from S3 from db-01 to db-02
   • Updated wal-g and patched it to continue from the partial restore
   • Configured a restore_command wrapper that fetched WALs from S3 and the local files, and replayed them in standby mode
   • Promoted db-02 to primary and restarted Synapse
   • Ran backups against the primary while no replica existed
   • Rebuilt db-01 as a replica from a base backup of db-02

   Action items

   • Run pg_repack to reclaim free space
   • Take incremental backups more often, since they restore faster than WAL replay
   • Reduce Synapse's storage requirements
   • Consider changing the terminal background colour by the database's role, for example by watching for the standby.signal file
   • Consider wrapper scripts around sensitive commands, such as an alias for rm, and automating operations such as starting a base backup from primary to secondary
   • Shorten the recovery time, with infrastructure from service providers
   • Upgrade wal-g on all servers and repeat disaster recovery testing with it
   • Explore ZFS local snapshots at the next hardware refresh
   • Make status.matrix.org the source of truth during incidents and keep social media updated

   Lessons

   Most of the difficulty came from the size of the database: restoring 51TB and replaying WALs took far longer than hoped, and incremental backups restore much faster than WAL replay. Running a destructive command on the wrong server was the turning point, and Matrix.org looks for better tooling and processes rather than blaming the individual; working as a trio did not catch the mistake, and server names that suggest a role caused confusion because a primary and a secondary swap roles. The recovery lost no data, but the recovery time needs to be shorter, and the wal-g versions on the disaster recovery server and the production servers had drifted apart.

   View case
3. IL-0526 OpenStreetMap.org went offline, then read-only, for 56 hours after HE.net routing equipment cut both fibre uplinks

   OpenStreetMap Foundation2024

   Symptoms

   At 04:01 UTC on 15 December 2024, StatusCake monitoring sent SMS alerts that OpenStreetMap.org was unreachable. The website and the mapping API had gone completely offline at 03:53 UTC.

   Network unreachableService outage

   Trigger

   HE.net's routing equipment in Amsterdam failed at 03:53 UTC on 15 December 2024.

   Root cause

   Dependency failure

   Component

   ISP and transit networksDatabase replication

   Contributing factor

   Single point of failureNo runbook

   Detection

   StatusCake monitoring detected the outage at 04:01 UTC and sent SMS alerts, and the incident was reported to the operations team at once.

   Remediation

   • Activated a read-only backup instance of OpenStreetMap.org in Dublin
   • Brought a new Equinix Internet link into service in Amsterdam, bypassing HE.net and restoring full functionality
   • HE.net replaced its routing hardware and restored its service, and routing through HE.net resumed

   Action items

   • Move to a multi-ISP architecture so another provider can take over when one fails
   • Research backup ISPs for the Amsterdam servers, including BGP and keeping HE.net as a fallback
   • Add deeper route and latency monitoring and an automated escalation protocol
   • Hold failover drills for switching between ISPs and keep replication up to date
   • Agree SLAs with HE.net and Equinix Internet with defined escalation procedures and response times
   • Prepare the 4G out-of-band system as a manual fallback uplink that can sync Postgres replication data, and upgrade its hardware
   • Document outage communication procedures and delegate communication during outages
   • Investigate decoupling the OAuth service so authentication keeps working while the database is read-only

   Lessons

   Relying on HE.net alone left the Amsterdam servers with a single point of failure, so OpenStreetMap will keep multiple upstream providers. The 4G out-of-band link proved invaluable and should be able to carry data synchronisation. Resumed replication would have restored editing faster than the read-only backup. Frequent contact with the ISP gave no useful estimates, so the SLA needs to be better understood. Alerting on upstream connectivity needs refining, and a read-only OpenStreetMap.org also blocked logins to services that were not affected, because it could not save new tokens.

   View case
4. IL-0374 DigitalOcean creates and management actions failed globally when its service discovery datastore lost quorum

   DigitalOcean2024

   Symptoms

   From 16:30 UTC on 5 August 2024, DigitalOcean's internal services could not discover the other services they depended on. Creates, event processing and the management of DigitalOcean products failed in all regions through the Cloud Control Panel and the API, while existing customer resources such as Droplets stayed online.

   Service outageCluster or node provisioning stall

   Trigger

   During the rollout of a new datastore cluster in a new datacenter, a replication configuration was deployed against the datastore behind internal service discovery that gave the new datacenter, which held zero keys, 100% ownership of all keys.

   Root cause

   Configuration change

   Component

   Coordination and key-value storeService discoveryDatabase replication

   Contributing factor

   Excessive blast radius

   Detection

   Service discovery unavailability was first reported internally at 16:35 UTC, five minutes after the rollout of the new datastore cluster began.

   Remediation

   • Reverted the replication configuration and ran it against the region that had become the datastore owner
   • Applied the configuration by hand on the nodes of lagging regions and ran local data repairs
   • Cleaned up failed events and processed pending events that had not timed out

   Action items

   • Perform this kind of datastore extension inside a declared maintenance window, announced on the status page
   • Update the process documentation for this type of deployment, stage by stage
   • Automate the manual steps used in this deployment to reduce human error
   • Evaluate the internal datastore topology and whether to regionalise or layer it

   View case
5. IL-0308 Most RevenueCat backend systems failed after cutover to a new Aurora PostgreSQL 14 cluster as queries became very slow

   RevenueCat2022

   Symptoms

   At 9:34 UTC on November 23, 2022, RevenueCat started moving traffic from its Aurora PostgreSQL 10.x cluster to a new Aurora PostgreSQL 14.x cluster. By 9:39 UTC all traffic was on the new cluster, latency was bad, and the RevenueCat dashboard and the endpoints used by the RevenueCat SDK stopped working.

   Database overloadHigh latencyService outageConnection pool exhaustion

   Trigger

   At 9:34 UTC production traffic was switched from the Aurora PostgreSQL 10.x cluster to the new Aurora PostgreSQL 14.x cluster, before ANALYZE had run on its largest tables.

   Root cause

   Breaking change in an upgrade

   Component

   DatabasePostgreSQL vacuumDatabase replicationDatabase proxy and connection poolerQuery planner and table statistics

   Contributing factor

   Unrepresentative stagingUnverified assumptionAutomation amplificationMonitoring blind spot

   Detection

   At 9:39 UTC, with all traffic on the new cluster, latency looked bad and the infrastructure team began to investigate.

   Remediation

   • Added a statement timeout to the database user so queries could not hold a connection too long
   • Shed 90% of traffic by returning 503s from the load balancer
   • Dropped indexes that the planner was using wrongly and that were not needed
   • Stopped accepting Apple webhooks for a time
   • Rolled out a release without New Relic to stop its EXPLAIN queries
   • Ran ANALYZE by hand on the biggest tables
   • Raised the values of the affected primary key sequences with a prepared script
   • Stopped the integration event dispatcher by hand
   • Replayed the webhooks that were dispatched but not recorded
   • Restored integration event dispatching, ETLs and other non-web server jobs

   Action items

   • Consider change freezes during business-critical periods
   • Explain better how the SDK behaves during backend outages
   • Tell customers about maintenance and outages in the developer dashboard and by direct email to technical contacts
   • Replay a subset of write traffic in migration dry runs to mimic the cutover
   • Move costly processes to asynchronous work to keep the API fast and stable
   • Re-evaluate table indexes, including unused ones the database may choose for queries
   • Show the load of each endpoint and system in the database dashboards
   • Involve support and customer communications staff during outages

   Lessons

   The new cluster had not been analyzed, and the tests missed it because the test cluster had been up long enough to gather statistics. Dry runs replayed no write traffic, on the belief that writes are simpler than reads, and the team assumed sequence state was replicated with the data. Rolling back at the start would have been better than fixing forward. The migration, timed to finish before Thanksgiving and Black Friday, should have been delayed until the week after, because the US had entered a sensitive period and part of the US support team was already on holiday.

   View case

By root cause8