Cases2 cases

1 to 2 of 2

1. IL-0390 PostHog US Cloud event ingestion slowed to delays of up to 2 days as writes to the Persons Postgres table stalled

   PostHog2025

   Symptoms

   From 11 November 2025 at 16:02 UTC, PostHog's event ingestion pipeline for US Cloud became severely degraded, and events for all customers in the US region were processed with delays from 10 minutes up to 2 days. Writes to the Postgres database that stores Person information could take multiple seconds per update, and the pipeline slowed until no standard scaling option helped.

   Throughput degradationHigh latency

   Trigger

   The 32-bit OID space used to track TOASTed values of the Persons table's properties column, a JSONb field that often exceeds the 2KB threshold for out-of-line storage, was nearly exhausted.

   Root cause

   Capacity exhaustion

   Component

   DatabaseMessage broker

   Contributing factor

   Unknown limitMonitoring blind spotNo runbook

   Remediation

   • Created a new partitioned Persons table with a fresh TOAST table and OID space
   • Added database triggers to keep the old and new Persons tables in sync for live writes
   • Copied the existing Person data from the old table to the new one with a backfill script
   • Changed ingestion and application code to write to the new table and read from it first, with fallback to the old table
   • Deployed the web app changes to production under manual control rather than a fully automated rollout
   • Scaled ingestion workers to process the accumulated backlog
   • Paused ingestion when the MSK local disk reached about 85%
   • Reduced the local retention of the affected Kafka topic
   • Resumed ingestion after confirming there was enough disk headroom
   • Moved the backfill into Dagster to finish it over the weekend

   Action items

   • Add metrics and alerts for heavy disk read amplification per write
   • Add metrics and alerts for TOAST table statistics
   • Add metrics and alerts for other Postgres engine limits that matter at large scale
   • Write runbooks for engine-level Postgres limits, with decision trees for migrating versus repairing in place
   • Create customer communication runbooks with a clear escalation path and redundancies
   • Evaluate other data stores for the persons database
   • Complete the reconsolidation on the new Persons tables
   • Improve the backfill tooling and the safeguards for Kafka tiered storage
   • Design and operate Person-like tables so they avoid similar limits

   Lessons

   TOAST OID exhaustion degrades writes long before the hard limit of about 4 billion OIDs, it is rare even at large scale, and standard dashboards for CPU, memory, IOPS and lock contention did not immediately point at it, so diagnosis took a day and a half without monitoring of TOAST growth, OID usage or disk reads per write. Many core features depended on timely updates to the persons table, so its slowdown reached a wide part of the product. The first backfill approach lacked visibility for a long migration, the Kafka tiered storage settings and alerts were not tuned for a large backlog, and the people who normally handle customer communications were all unavailable.

   View case
2. IL-0334 Instapaper's RDS MySQL bookmarks table hit the 2TB ext3 file size limit, inserts failed and the service went down

   Instapaper2017

   Symptoms

   On Wednesday, February 9, 2017 at 12:30PM PT, inserts of new entries into the bookmarks table of Instapaper's MySQL database on Amazon RDS started to fail with an error. The bookmarks table, which stores the articles Instapaper users save, had grown past 2TB, and the Instapaper service was down until Thursday, February 10 at 7:30pm PT.

   Writes rejectedService outage

   Trigger

   At 12:30PM PT on February 9 the bookmarks table grew past the 2TB file size limit of the instance's filesystem.

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase backups and snapshotsDatabase replication

   Contributing factor

   Unknown limitMonitoring blind spotUntested backupNo runbookLimited debug access

   Remediation

   • Dumped the production database to a 5.5TB RAID-0 disk on an i2.8xlarge instance with Pinterest's SRE team
   • Launched a temporary production instance with limited access to archives
   • Set up an Aurora read replica of the old database as one recovery path
   • Imported the data into a new MySQL RDS instance without secondary indexes, then built the indexes
   • Had an Amazon engineer rsync the data from the ext3 filesystem to an ext4 filesystem mounted on the failed database
   • Synced the new ext4 database with the temporary production database through row-based replication
   • Promoted the new database to master and pointed the application at it

   Action items

   • Define a workflow that escalates system-wide Instapaper outages immediately to Pinterest's SRE team
   • Test MySQL backups every month instead of every three months
   • Put a process in place with Pinterest's SRE team to recover from this type of failure

   Lessons

   A filesystem limit that Instapaper did not know about and could not see made both the production database and all its backups useless, and the only way out was to restore the data onto a new filesystem. Because the hosted database could only be reached through MySQL, filesystem-level fixes such as rsync needed Amazon engineers. Even with a perfect response, rebuilding the database would have taken at least 10 hours. The lack of a disaster recovery plan for this kind of failure, and the estimate of the dump time from row counts, made the downtime and recovery longer than necessary, and the response could have drawn better on Pinterest's and AWS's people. The new escalation workflow and monthly backup tests would not have prevented this failure but would speed up the response to a future one.

   View case

By root cause1