Incident Library
Applied filters Root causeSoftware defectOrganisationFigmaClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0486 Figma's PostgreSQL 9 database on RDS slowed under high write IOPS and lock contention until the site was unavailable
    Figma2020
    Symptoms

    On January 22, 2020 at 9:45 AM PST, Figma's database metrics showed rising write IOPS and lock contention, while database CPU was lower than normal. An automated alert followed at 10:00 AM, and by 11:42 AM enough API requests had queued up behind the database that Figma became unavailable for some users.

    Lock contention or deadlockDatabase overloadService outage
    Trigger

    An automatic ANALYZE on one of the tables in the query finished at exactly the moment the incident began. The new statistics pushed the planner's row estimate for an inner subquery past 200 rows, and the plan for the whole query switched to an estimate of over 20 million rows.

    Root cause
    Software defect
    Component
    DatabasePostgreSQL vacuum
    Contributing factor
    Overlapping failuresHarmful manual action
    Detection

    Database metrics showed higher write IOPS and lock contention at 9:45 AM PST, and an automated alert at 10:00 AM started the investigation.

    Remediation
    • Cancelled several inessential queries and increased the allocated IOPS
    • Cancelled autovacuum statements running on the largest tables
    • Changed autovacuum_freeze_max_age and restarted the database to stop aggressive autovacuuming
    • Took the service down for emergency maintenance and upgraded the database to PostgreSQL 11
    Action items
    • Revisit transaction timeouts and limits so long-running and expensive queries are terminated
    • Improve metrics, monitoring and database settings for autovacuuming, locking and query performance
    • Audit queries for unnecessary or excessive joins and subqueries that could trip up the query planner
    • Bring the performance tools of the upgraded PostgreSQL into the incident response plans
    • Audit the structure and access patterns of data stored in PostgreSQL and consider moving some of it to datastores suited to those patterns
    • Improve in-app messaging about unexpected maintenance windows
    • Move towards online database upgrades that do not need downtime
    • Let users keep working on already open files safely while offline, even after closing the browser
    Lessons

    Having upgraded staging to PostgreSQL 11, tested it for months and written a production upgrade plan let the team upgrade under poor conditions with confidence. A routine statistics refresh can flip a query to a catastrophic plan, and the same query's plan can differ widely over time, so comparing plans from snapshots taken at different dates exposed the cause.

    View case
Check your cluster for this

This case involves PostgreSQL vacuum and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
PostgreSQL vacuumDatabase
Contributing factor
Harmful manual actionOverlapping failures
Organisation
Figma
  1. IL-0486 Figma's PostgreSQL 9 database on RDS slowed under high write IOPS and lock contention until the site was unavailable
    Figma2020