Cases3 cases

1 to 3 of 3

1. IL-0368 PlanetScale's control plane went down and some database branches failed queries during the AWS us-east-1 outage

   PlanetScale2025

   Symptoms

   At 7:13 UTC on 20 October 2025, PlanetScale's continuous tests in production started failing because they could not create new database branches. The service that creates, resizes and configures database branches was unavailable, a near-total control plane outage, while existing database branches kept their capacity and connectivity.

   Service outageScale-up failureNetwork unreachableHigh error rateReplication lagDNS resolution failures

   Trigger

   The incident started with a DNS misconfiguration at one of PlanetScale's service providers. An Amazon DynamoDB outage impacted AWS STS, and several hours of capacity constraints and network instability in AWS us-east-1 followed.

   Root cause

   Cloud provider outage

   Component

   DatabaseDatabase replicationDatabase proxy and connection poolerObject storage

   Contributing factor

   –

   Detection

   PlanetScale's continuous testing in production started failing and alerted engineers at 7:13 UTC. At 10:05 UTC an alert fired for a Kubernetes operator in a large single-tenant customer installation exhausting its resources.

   Remediation

   • Disallowed new databases in us-east-1 and made us-east-2 the default region for new databases
   • Delayed new backups and cancelled pending backups that were waiting for an EC2 instance
   • Advised PlanetScale Managed customers with vtgate autoscaling to shed load
   • Paused draining and terminating EC2 instances older than 30 days
   • Kept vacant EC2 instances for reuse instead of terminating them
   • Bin-packed vtgate processes more tightly for customers with autoscaling
   • Sent manual reparent requests to move primaries to healthier availability zones
   • Restarted the edge load balancer and vtgate processes that did not recover after the partitions healed

   Action items

   • Understand and become resilient to the failure modes of SaaS dependencies for CI/CD, SSO, web application hosting and incident communication
   • Investigate more ambitious ways to reduce runtime dependence on internal and AWS services
   • Look into using all six us-east-1 availability zones to tolerate zonal outages and partitions

   Lessons

   Keeping the control plane separate from the data plane meant the control plane outage did not affect customer databases, and redundancy, automated failover and zonal traffic routing limited the effect of the network partitions. Network partitions are hard to reason about, test and tolerate: with three availability zones a system survives losing one zone only if the other two can still reach each other reliably.

   View case
2. IL-0526 OpenStreetMap.org went offline, then read-only, for 56 hours after HE.net routing equipment cut both fibre uplinks

   OpenStreetMap Foundation2024

   Symptoms

   At 04:01 UTC on 15 December 2024, StatusCake monitoring sent SMS alerts that OpenStreetMap.org was unreachable. The website and the mapping API had gone completely offline at 03:53 UTC.

   Network unreachableService outage

   Trigger

   HE.net's routing equipment in Amsterdam failed at 03:53 UTC on 15 December 2024.

   Root cause

   Dependency failure

   Component

   ISP and transit networksDatabase replication

   Contributing factor

   Single point of failureNo runbook

   Detection

   StatusCake monitoring detected the outage at 04:01 UTC and sent SMS alerts, and the incident was reported to the operations team at once.

   Remediation

   • Activated a read-only backup instance of OpenStreetMap.org in Dublin
   • Brought a new Equinix Internet link into service in Amsterdam, bypassing HE.net and restoring full functionality
   • HE.net replaced its routing hardware and restored its service, and routing through HE.net resumed

   Action items

   • Move to a multi-ISP architecture so another provider can take over when one fails
   • Research backup ISPs for the Amsterdam servers, including BGP and keeping HE.net as a fallback
   • Add deeper route and latency monitoring and an automated escalation protocol
   • Hold failover drills for switching between ISPs and keep replication up to date
   • Agree SLAs with HE.net and Equinix Internet with defined escalation procedures and response times
   • Prepare the 4G out-of-band system as a manual fallback uplink that can sync Postgres replication data, and upgrade its hardware
   • Document outage communication procedures and delegate communication during outages
   • Investigate decoupling the OAuth service so authentication keeps working while the database is read-only

   Lessons

   Relying on HE.net alone left the Amsterdam servers with a single point of failure, so OpenStreetMap will keep multiple upstream providers. The 4G out-of-band link proved invaluable and should be able to carry data synchronisation. Resumed replication would have restored editing faster than the read-only backup. Frequent contact with the ISP gave no useful estimates, so the SLA needs to be better understood. Alerting on upstream connectivity needs refining, and a read-only OpenStreetMap.org also blocked logins to services that were not affected, because it could not save new tokens.

   View case
3. IL-0431 During Mackerel's move to AWS, PostgreSQL failed over to a replica that had stopped replicating, losing an hour of data

   Hatena2017

   Symptoms

   On 7 August 2017, while checking the application after the maintenance that moved Mackerel from an on-premises data center to AWS, the team found that data updated during a specific time window was missing. Everything written to the PostgreSQL database between 14:30 and 15:30 JST was gone.

   Data lossReplication lagNetwork unreachable

   Trigger

   One hour after the application was switched to the newly promoted primary db001, db001 lost network connectivity and the database failed over to the standby db002.

   Root cause

   Operator error

   Component

   DatabaseDatabase replicationDatabase failover manager

   Contributing factor

   Overlapping failuresUnreviewed defaultsUnrepresentative stagingAutomation amplification

   Detection

   The team found the missing data while checking how the application behaved after the migration work.

   Remediation

   • Tried to restore the lost updates by hand from the table data left on db001 and gave up
   • Tried point-in-time recovery from the base backup and WAL and gave up
   • Built a new replica db003 with db002 as primary to restore redundancy
   • Kept db001 in maintenance state for investigation
   • Watched db002 for some time to confirm the problem did not recur, then completed the migration

   Action items

   • Change the PostgreSQL replica creation script to set recovery_target_timeline = 'latest' in recovery.conf by default
   • Add a check to a server configuration test tool such as Serverspec that recovery.conf contains recovery_target_timeline = 'latest'

   Lessons

   A setting needed only for a rare layout, such as a grandchild replica during a migration, was applied by hand through a script option and was forgotten. Because recovery_target_timeline = 'latest' is harmless on ordinary standbys too, Hatena now sets it by default whenever a replica is created. The staging rehearsal used the same layout and steps but had the setting right, so it could not catch the mistake. Hatena notes that a similar layout may be used again for a future PostgreSQL upgrade or a move to another region.

   View case

By root cause3