Incident Library
Applied filters ComponentObject storageSymptomReplication lagClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0368 PlanetScale's control plane went down and some database branches failed queries during the AWS us-east-1 outage
    PlanetScale2025
    Symptoms

    At 7:13 UTC on 20 October 2025, PlanetScale's continuous tests in production started failing because they could not create new database branches. The service that creates, resizes and configures database branches was unavailable, a near-total control plane outage, while existing database branches kept their capacity and connectivity.

    Service outageScale-up failureNetwork unreachableHigh error rateReplication lagDNS resolution failures
    Trigger

    The incident started with a DNS misconfiguration at one of PlanetScale's service providers. An Amazon DynamoDB outage impacted AWS STS, and several hours of capacity constraints and network instability in AWS us-east-1 followed.

    Root cause
    Cloud provider outage
    Component
    DatabaseDatabase replicationDatabase proxy and connection poolerObject storage
    Contributing factor
    –
    Detection

    PlanetScale's continuous testing in production started failing and alerted engineers at 7:13 UTC. At 10:05 UTC an alert fired for a Kubernetes operator in a large single-tenant customer installation exhausting its resources.

    Remediation
    • Disallowed new databases in us-east-1 and made us-east-2 the default region for new databases
    • Delayed new backups and cancelled pending backups that were waiting for an EC2 instance
    • Advised PlanetScale Managed customers with vtgate autoscaling to shed load
    • Paused draining and terminating EC2 instances older than 30 days
    • Kept vacant EC2 instances for reuse instead of terminating them
    • Bin-packed vtgate processes more tightly for customers with autoscaling
    • Sent manual reparent requests to move primaries to healthier availability zones
    • Restarted the edge load balancer and vtgate processes that did not recover after the partitions healed
    Action items
    • Understand and become resilient to the failure modes of SaaS dependencies for CI/CD, SSO, web application hosting and incident communication
    • Investigate more ambitious ways to reduce runtime dependence on internal and AWS services
    • Look into using all six us-east-1 availability zones to tolerate zonal outages and partitions
    Lessons

    Keeping the control plane separate from the data plane meant the control plane outage did not affect customer databases, and redundancy, automated failover and zonal traffic routing limited the effect of the network partitions. Network partitions are hard to reason about, test and tolerate: with three availability zones a system survives losing one zone only if the other two can still reach each other reliably.

    View case
Check your cluster for this

This case involves Database and Database proxy and connection pooler. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Cloud provider outage1 case

Component
DatabaseDatabase proxy and connection poolerDatabase replicationObject storage
Contributing factor
–
Organisation
PlanetScale
  1. IL-0368 PlanetScale's control plane went down and some database branches failed queries during the AWS us-east-1 outage
    PlanetScale2025