Cases33 cases

13 to 17 of 33

1. IL-0374 DigitalOcean creates and management actions failed globally when its service discovery datastore lost quorum

   DigitalOcean2024

   Symptoms

   From 16:30 UTC on 5 August 2024, DigitalOcean's internal services could not discover the other services they depended on. Creates, event processing and the management of DigitalOcean products failed in all regions through the Cloud Control Panel and the API, while existing customer resources such as Droplets stayed online.

   Service outageCluster or node provisioning stall

   Trigger

   During the rollout of a new datastore cluster in a new datacenter, a replication configuration was deployed against the datastore behind internal service discovery that gave the new datacenter, which held zero keys, 100% ownership of all keys.

   Root cause

   Configuration change

   Component

   Coordination and key-value storeService discoveryDatabase replication

   Contributing factor

   Excessive blast radius

   Detection

   Service discovery unavailability was first reported internally at 16:35 UTC, five minutes after the rollout of the new datastore cluster began.

   Remediation

   • Reverted the replication configuration and ran it against the region that had become the datastore owner
   • Applied the configuration by hand on the nodes of lagging regions and ran local data repairs
   • Cleaned up failed events and processed pending events that had not timed out

   Action items

   • Perform this kind of datastore extension inside a declared maintenance window, announced on the status page
   • Update the process documentation for this type of deployment, stage by stage
   • Automate the manual steps used in this deployment to reduce human error
   • Evaluate the internal datastore topology and whether to regionalise or layer it

   View case
2. IL-0488 Kit's RDS MySQL write latency rose 100x and running threads jumped from 9 to 900, backing up Sidekiq job queues

   Kit2022

   Symptoms

   At 5:20 am EST on December 24, 2022, Kit's infrastructure team got its first page because the number of MySQL threads running on its RDS MySQL primary in us-east-2 had jumped 100x, from 9 to 900, at a time of day when Kit's traffic is very low. Sidekiq queues were backing up.

   Database overloadThroughput degradation

   Trigger

   An EBS interruption on the standby of the Multi-AZ RDS instance, reported by AWS as an LSE at 02:12:47 UTC on 2022-12-24, started a replaceSecondaryHostAndVolumesTask that replaced the secondary host and volumes.

   Root cause

   Cloud provider outage

   Component

   DatabaseCloud block storageDatabase replication

   Contributing factor

   Ignored early signal

   Detection

   A page fired at 5:20 am EST because MySQL threads running jumped from 9 to 900. An AWS RSS feed alert in Slack about degraded EC2 and EBS in use2-az1 had arrived the evening before and was not acted on.

   Remediation

   • Scaled the RDS storage from 12.5 TB to 13.75 TB, which put the instance through "optimizing storage" and brought write latency back to 1-2 ms

   Lessons

   Multi-AZ RDS keeps data consistent and durable, but synchronous replication means each write also waits for the standby, so a degraded EBS volume on the standby while it is being replaced slows writes on the primary. Large databases recover more slowly from such events than small ones. A single-AZ instance would not have been hit by a standby problem, but would have suffered hours of downtime if its own zone failed, and that trade-off has to be weighed when deploying production RDS instances.

   View case
3. IL-0308 Most RevenueCat backend systems failed after cutover to a new Aurora PostgreSQL 14 cluster as queries became very slow

   RevenueCat2022

   Symptoms

   At 9:34 UTC on November 23, 2022, RevenueCat started moving traffic from its Aurora PostgreSQL 10.x cluster to a new Aurora PostgreSQL 14.x cluster. By 9:39 UTC all traffic was on the new cluster, latency was bad, and the RevenueCat dashboard and the endpoints used by the RevenueCat SDK stopped working.

   Database overloadHigh latencyService outageConnection pool exhaustion

   Trigger

   At 9:34 UTC production traffic was switched from the Aurora PostgreSQL 10.x cluster to the new Aurora PostgreSQL 14.x cluster, before ANALYZE had run on its largest tables.

   Root cause

   Breaking change in an upgrade

   Component

   DatabasePostgreSQL vacuumDatabase replicationDatabase proxy and connection poolerQuery planner and table statistics

   Contributing factor

   Unrepresentative stagingUnverified assumptionAutomation amplificationMonitoring blind spot

   Detection

   At 9:39 UTC, with all traffic on the new cluster, latency looked bad and the infrastructure team began to investigate.

   Remediation

   • Added a statement timeout to the database user so queries could not hold a connection too long
   • Shed 90% of traffic by returning 503s from the load balancer
   • Dropped indexes that the planner was using wrongly and that were not needed
   • Stopped accepting Apple webhooks for a time
   • Rolled out a release without New Relic to stop its EXPLAIN queries
   • Ran ANALYZE by hand on the biggest tables
   • Raised the values of the affected primary key sequences with a prepared script
   • Stopped the integration event dispatcher by hand
   • Replayed the webhooks that were dispatched but not recorded
   • Restored integration event dispatching, ETLs and other non-web server jobs

   Action items

   • Consider change freezes during business-critical periods
   • Explain better how the SDK behaves during backend outages
   • Tell customers about maintenance and outages in the developer dashboard and by direct email to technical contacts
   • Replay a subset of write traffic in migration dry runs to mimic the cutover
   • Move costly processes to asynchronous work to keep the API fast and stable
   • Re-evaluate table indexes, including unused ones the database may choose for queries
   • Show the load of each endpoint and system in the database dashboards
   • Involve support and customer communications staff during outages

   Lessons

   The new cluster had not been analyzed, and the tests missed it because the test cluster had been up long enough to gather statistics. Dry runs replayed no write traffic, on the belief that writes are simpler than reads, and the team assumed sequence state was replicated with the data. Rolling back at the start would have been better than fixing forward. The migration, timed to finish before Thanksgiving and Black Friday, should have been delayed until the week after, because the US had entered a sensitive period and part of the US support team was already on holiday.

   View case
4. IL-0311 BattleMetrics' primary PostgreSQL went read-only at transaction ID wraparound, breaking the website and RCON updates

   BattleMetrics2022

   Symptoms

   At 03:23 UTC on 27 March 2022, BattleMetrics' primary PostgreSQL database, which holds all account, server and player information, went into read-only mode. Users first saw the website fail to load, and soon RCON updates and queries began to fail as well.

   Writes rejectedService outage

   Trigger

   A corruption of the index on the player identifiers table, which BattleMetrics' logs place at about 26 February 2022, made every vacuum job fail, so transaction IDs kept climbing until Postgres stopped accepting writes on 27 March.

   Root cause

   Data loss or corruption

   Component

   DatabasePostgreSQL vacuumDatabase replicationDatabase backups and snapshots

   Contributing factor

   Monitoring blind spotOverlapping failuresNo headroom

   Detection

   Users failed to load the website and reported it as their API keys expired, then RCON updates and queries began to fail, and the team found that the main database no longer accepted writes.

   Remediation

   • Shut the database down and dropped the corrupted player identifiers index in single-user mode
   • Ran the vacuum again, which succeeded and let the database accept writes
   • Rebuilt the player identifier index, which took about two hours
   • Shut down the query workers and gave RCON updates priority
   • Pointed services at the new Postgres primary
   • Bootstrapped new replicas one after another to take load off the primary
   • Restarted queries with throttling, prioritised by server population and rank, after RCON updates had caught up

   Action items

   • Monitor errors in the Postgres logs
   • Monitor and alert on transaction IDs
   • Set up a logical replica in addition to physical replication
   • Build a faster network to speed up recovery from failure

   Lessons

   BattleMetrics names skipping health checks on transaction ID size as its critical mistake: the index corruption silently broke vacuum for a month, and without monitoring of transaction IDs nobody saw the database approach wraparound. Because the corruption had been replicated to the replicas, failing over did not help, and the backup servers and downloadable backups carried the same corruption. The team also found that, contrary to the error message, a full vacuum on a server in wraparound protection does not need single-user mode, and chose to run a slow, delayed service over no service while replicas were rebuilt.

   View case
5. IL-0240 Cloudflare API requests failed and the dashboard slowed for hours after the authentication database lost its replicas

   Cloudflare2020

   Symptoms

   On 2 November 2020 the Cloudflare API and dashboard were degraded for six hours and 33 minutes. The success rate of API queries dropped at times to 75%, and the dashboard was up to 80 times slower than normal, while Cloudflare's edge network kept working.

   High error rateHigh latencyDatabase overloadWrites rejectedPacket loss

   Trigger

   At 14:43 UTC a network switch in a control plane data center partly failed: it kept LACP and BGP running but stopped forwarding some packets, and it recovered on its own six minutes later.

   Root cause

   DNS or network fault

   Component

   NIC, cable and switch port linkCoordination and key-value storeDatabase failover managerDatabaseDatabase replication

   Contributing factor

   Automation amplificationNo headroomHarmful manual action

   Detection

   At 14:43 UTC alerts fired because the misbehaving switch was unreachable to pings. Engineers then saw that the primary authentication database was overwhelmed.

   Remediation

   • Shed load from the primary authentication database by slowing SSL certificate pushes to the edge, emails and other features
   • Moved API traffic that could use read replicas to the secondary data center by hand
   • Sent authentication calls fully back to the primary data center when the dashboard got worse
   • Rebuilt the first database replica, which put itself back into service at 21:20 UTC and returned performance to normal
   • Ramped the turned-down services back up so asynchronous processing could catch up

   Action items

   • Remove the cluster management limitation that requires rebuilding all replicas after a primary is promoted
   • Redesign the user sessions system so dashboard traffic can be steered between data centers
   • Change the auto-remediation configuration so a database replica is not promoted so quickly
   • Tune each system involved to be more resilient to failures of its dependencies

   Lessons

   Every system in the chain had redundancy and none failed completely; each only degraded, which made the chain of events much harder to model and foresee. Automating replica promotion so that it could trigger on a minute's notice had been a point of pride, but for at least one database a fast promotion can do more harm than the failure it answers. Most general-purpose cluster managers use Raft or Paxos rather than Byzantine fault tolerant protocols, accepting a rare failure mode in exchange for a protocol that is simpler to implement correctly and debug.

   View case

By root cause11