Cases7 cases

4 to 7 of 7

1. IL-0315 Railway's backend API went down when an index build locked a billion-row Postgres table and used up every connection

   Railway2025

   Symptoms

   At 18:36 UTC on 28 October 2025, two minutes after a database change went live in production, Railway's internal monitoring alerted and paged the engineering team. The dashboard could not be opened, CLI operations that use the API, such as railway up, failed, GitHub-based deployments were delayed because webhook processing failed, and all Railway Public API operations were disrupted.

   Service outageLock contention or deadlockDatabase overload

   Trigger

   A routine schema change to the backend's Postgres database, introduced at 18:23 UTC and live in production at 18:34 UTC, that added a new column with an index without the CONCURRENTLY option.

   Root cause

   Faulty deployment

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   No configuration testsLimited debug access

   Detection

   Internal monitoring alerts paged the engineering team at 18:36 UTC, and the incident was declared at 18:41 UTC.

   Remediation

   • The index migration finished after about 30 minutes and released the table lock, and the queued operations and connection pools recovered

   Action items

   • Enforce CONCURRENTLY for all index creation in CI and block pull requests that do not use it
   • Adjust PgBouncer pool limits so they cannot overwhelm the Postgres instance's capacity
   • Configure database user connection limits so administrative access stays possible during incidents

   View case
2. IL-0368 PlanetScale's control plane went down and some database branches failed queries during the AWS us-east-1 outage

   PlanetScale2025

   Symptoms

   At 7:13 UTC on 20 October 2025, PlanetScale's continuous tests in production started failing because they could not create new database branches. The service that creates, resizes and configures database branches was unavailable, a near-total control plane outage, while existing database branches kept their capacity and connectivity.

   Service outageScale-up failureNetwork unreachableHigh error rateReplication lagDNS resolution failures

   Trigger

   The incident started with a DNS misconfiguration at one of PlanetScale's service providers. An Amazon DynamoDB outage impacted AWS STS, and several hours of capacity constraints and network instability in AWS us-east-1 followed.

   Root cause

   Cloud provider outage

   Component

   DatabaseDatabase replicationDatabase proxy and connection poolerObject storage

   Contributing factor

   –

   Detection

   PlanetScale's continuous testing in production started failing and alerted engineers at 7:13 UTC. At 10:05 UTC an alert fired for a Kubernetes operator in a large single-tenant customer installation exhausting its resources.

   Remediation

   • Disallowed new databases in us-east-1 and made us-east-2 the default region for new databases
   • Delayed new backups and cancelled pending backups that were waiting for an EC2 instance
   • Advised PlanetScale Managed customers with vtgate autoscaling to shed load
   • Paused draining and terminating EC2 instances older than 30 days
   • Kept vacant EC2 instances for reuse instead of terminating them
   • Bin-packed vtgate processes more tightly for customers with autoscaling
   • Sent manual reparent requests to move primaries to healthier availability zones
   • Restarted the edge load balancer and vtgate processes that did not recover after the partitions healed

   Action items

   • Understand and become resilient to the failure modes of SaaS dependencies for CI/CD, SSO, web application hosting and incident communication
   • Investigate more ambitious ways to reduce runtime dependence on internal and AWS services
   • Look into using all six us-east-1 availability zones to tolerate zonal outages and partitions

   Lessons

   Keeping the control plane separate from the data plane meant the control plane outage did not affect customer databases, and redundancy, automated failover and zonal traffic routing limited the effect of the network partitions. Network partitions are hard to reason about, test and tolerate: with three availability zones a system survives losing one zone only if the other two can still reach each other reliably.

   View case
3. IL-0312 Gadget's main Postgres did not start again after a restart to raise max_connections, and the platform went down

   Gadget2024

   Symptoms

   At 11:25 AM EST on 13 December 2024, Gadget restarted its main PostgreSQL server in place to raise max_connections. Postgres shut down but did not start again, and every request to Gadget applications failed, including frontend requests, GraphQL API requests, webhook processing and syncs.

   Service outageHigh latency

   Trigger

   An emergency in-place restart of the main Postgres server at 11:25 AM EST, decided at 11:00 AM, to raise max_connections, which Postgres cannot change without a full restart.

   Root cause

   Capacity exhaustion

   Component

   DatabaseLinux kernelDatabase proxy and connection pooler

   Contributing factor

   Unverified assumptionUnrepresentative stagingAutomation amplificationExcessive blast radius

   Detection

   The team carrying out the planned restart saw at 11:26 AM EST that Postgres had shut down and did not start again.

   Remediation

   • Restarted the database server with a hardware reset, which cleared the fragmented memory so Postgres could start
   • Provisioned almost 4x the compute after the autoscalers had scaled it down during the outage

   Action items

   • Use only the replica-failover approach for disruptive database operations such as restarts
   • Run a more aggressive capacity planning process before Black Friday / Cyber Monday, preparing for a 6x increase in load and proving it with load tests well in advance
   • Announce planned downtime ahead of time on the status page and community Discord, and make the operating procedures tell planned maintenance apart from service disruptions
   • Finish the database sharding project so apps can use different databases and configuration changes can go through canary and blue/green setups
   • Finish building a zero-downtime database failover system
   • Set much higher scaling floors on the compute autoscalers

   Lessons

   Gadget names three deeper factors. Choosing the in-place restart put the database into a state the team had not seen before and was not confident in. Capacity planning for Black Friday / Cyber Monday assumed a doubling of traffic instead of the almost 5x that came, so settings such as max_connections stopped fitting and had to be changed in an emergency instead of in a maintenance window practised in staging. And there was no way to test database configuration changes in a production-like environment: integration tests run against a synthetic Postgres database that never goes through the months of production load behind the memory fragmentation or the memory usage issues.

   View case
4. IL-0308 Most RevenueCat backend systems failed after cutover to a new Aurora PostgreSQL 14 cluster as queries became very slow

   RevenueCat2022

   Symptoms

   At 9:34 UTC on November 23, 2022, RevenueCat started moving traffic from its Aurora PostgreSQL 10.x cluster to a new Aurora PostgreSQL 14.x cluster. By 9:39 UTC all traffic was on the new cluster, latency was bad, and the RevenueCat dashboard and the endpoints used by the RevenueCat SDK stopped working.

   Database overloadHigh latencyService outageConnection pool exhaustion

   Trigger

   At 9:34 UTC production traffic was switched from the Aurora PostgreSQL 10.x cluster to the new Aurora PostgreSQL 14.x cluster, before ANALYZE had run on its largest tables.

   Root cause

   Breaking change in an upgrade

   Component

   DatabasePostgreSQL vacuumDatabase replicationDatabase proxy and connection poolerQuery planner and table statistics

   Contributing factor

   Unrepresentative stagingUnverified assumptionAutomation amplificationMonitoring blind spot

   Detection

   At 9:39 UTC, with all traffic on the new cluster, latency looked bad and the infrastructure team began to investigate.

   Remediation

   • Added a statement timeout to the database user so queries could not hold a connection too long
   • Shed 90% of traffic by returning 503s from the load balancer
   • Dropped indexes that the planner was using wrongly and that were not needed
   • Stopped accepting Apple webhooks for a time
   • Rolled out a release without New Relic to stop its EXPLAIN queries
   • Ran ANALYZE by hand on the biggest tables
   • Raised the values of the affected primary key sequences with a prepared script
   • Stopped the integration event dispatcher by hand
   • Replayed the webhooks that were dispatched but not recorded
   • Restored integration event dispatching, ETLs and other non-web server jobs

   Action items

   • Consider change freezes during business-critical periods
   • Explain better how the SDK behaves during backend outages
   • Tell customers about maintenance and outages in the developer dashboard and by direct email to technical contacts
   • Replay a subset of write traffic in migration dry runs to mimic the cutover
   • Move costly processes to asynchronous work to keep the API fast and stable
   • Re-evaluate table indexes, including unused ones the database may choose for queries
   • Show the load of each endpoint and system in the database dashboards
   • Involve support and customer communications staff during outages

   Lessons

   The new cluster had not been analyzed, and the tests missed it because the test cluster had been up long enough to gather statistics. Dry runs replayed no write traffic, on the belief that writes are simpler than reads, and the team assumed sequence state was replicated with the data. Rolling back at the start would have been better than fixing forward. The migration, timed to finish before Thanksgiving and Black Friday, should have been delayed until the week after, because the US had entered a sensitive period and part of the US support team was already on holiday.

   View case

By root cause5