Incident Library

IL-0368 PlanetScale · 2025-10-20awsdatabasenetworking

PlanetScale's control plane went down and some database branches failed queries during the AWS us-east-1 outage

The incident had two phases. In the first, the PlanetScale control plane service that creates, resizes and configures database branches, which is hosted in AWS us-east-1, was unavailable. That service depends on PlanetScale's internal secret-distribution service, which depends on Amazon S3, which depends on AWS STS, and AWS STS was impacted by the Amazon DynamoDB outage. The dashboard, hosted by a provider that also runs in us-east-1, was intermittently available, SSO logins failed and the status site could not be updated. No database branch lost capacity or connectivity. PlanetScale engineers investigated but took no corrective action, and service returned at 9:30 UTC once the upstream providers recovered.

In the second phase, from 10:05 UTC, new EC2 instances could not be launched in us-east-1, so branch creates and resizes stayed queued while existing MySQL and Postgres servers kept running. To need fewer new instances, engineers disallowed new databases in us-east-1 and made us-east-2 the default region, delayed new backups and cancelled pending ones that were waiting for an instance (a PlanetScale backup launches an extra replica that restores the previous backup and catches up on replication before taking the new one), asked PlanetScale Managed customers with vtgate autoscaling to shed load, paused the routine draining of instances older than 30 days, and kept vacant instances for reuse instead of terminating them. The most important change was to bin-pack vtgate processes more tightly than usual, closer to CPU capacity, for customers with autoscaling, so there was enough vtgate capacity for the US work day.

From about 14:30 UTC partial network partitions between availability zones made many queries fail for some customers, depending on which zones a branch used and whether its traffic crossed zones. Automated failover moved primaries to the majority side of the partitions, and engineers sent manual reparent requests to move primaries to healthier zones or next to the customer's application. The partitions healed gradually between about 18:30 and 19:30 UTC. PlanetScale's edge load balancer and some vtgate processes could not recover on their own because of how those processes had experienced the partition, and engineers restarted them. The incident commander declared the incident resolved at 20:32 UTC.

Severity
–
Scope
PlanetScale control plane (creating, resizing and configuring database branches), dashboard, SSO login and status page; queries, replication and query routing for some database branches in us-east-1
Users affected
–
Region
AWS us-east-1
Time to detect
–
Time to resolve
–

Symptoms

At 7:13 UTC on 20 October 2025, PlanetScale's continuous tests in production started failing because they could not create new database branches. The service that creates, resizes and configures database branches was unavailable, a near-total control plane outage, while existing database branches kept their capacity and connectivity.

During the same phase the PlanetScale dashboard was available only on and off, customers who sign in with SSO could not log in unless they were already logged in, and PlanetScale could not post updates to its status site, which was itself down for at least half an hour.

At 10:05 UTC an alert showed a Kubernetes operator in a large single-tenant customer installation using up all its resources, which usually means rising latency or control plane requests failing and being retried. Responders found that new EC2 instances could not be launched in us-east-1, so requests to create or resize database branches stayed queued. With the US East Coast about to start its Monday, some of the largest customers, who autoscale the vtgate component of their Vitess clusters on a daily cycle, were heading into their weekly peak traffic with less than half the vtgate capacity of the week before.

From about 14:30 UTC partial network partitions in us-east-1 made a significant share of some customers' queries fail. Some database servers were reachable from the Internet but could not talk across availability zones for query routing, replication or both. Some replicas started and reached container registries but could not replicate from their MySQL or Postgres primary. Some servers could not resolve internal DNS names, and others could not connect to the internal services those names resolved to.

Symptom Service outageScale-up failureNetwork unreachableHigh error rateReplication lagDNS resolution failures

Root cause

Root cause Cloud provider outage

Component DatabaseDatabase replicationDatabase proxy and connection poolerObject storage
Contributing factor –
Trigger
The incident started with a DNS misconfiguration at one of PlanetScale's service providers. An Amazon DynamoDB outage impacted AWS STS, and several hours of capacity constraints and network instability in AWS us-east-1 followed.

Detection

PlanetScale's continuous testing in production started failing and alerted engineers at 7:13 UTC. At 10:05 UTC an alert fired for a Kubernetes operator in a large single-tenant customer installation exhausting its resources.

Remediation

Action items

Lessons

Keeping the control plane separate from the data plane meant the control plane outage did not affect customer databases, and redundancy, automated failover and zonal traffic routing limited the effect of the network partitions. Network partitions are hard to reason about, test and tolerate: with three availability zones a system survives losing one zone only if the other two can still reach each other reliably.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Database replication, Database proxy and connection pooler among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.