Incident Library

IL-0390 PostHog · 2025-11-11databaseaws

PostHog US Cloud event ingestion slowed to delays of up to 2 days as writes to the Persons Postgres table stalled

PostgreSQL stores large column values (>2KB compressed) out of line in a TOAST table, and each such value gets a unique OID from a finite 32-bit space of about 4 billion values. PostHog's Persons table has a properties column that often exceeds that threshold, so it has many TOASTed values. As used OIDs approached the limit, Postgres had to read through long runs of used OIDs to find a free one for each INSERT or UPDATE of a TOASTed row, and writes to the Persons database kept waiting for OIDs. Write performance became severely degraded well before the hard limit, and event ingestion for US Cloud ground to a halt from 11 November 16:02 UTC.

The cause was identified on 12 November at 10:17pm UTC with AWS RDS engineers and external consultants, and the only option left was a migration to a new partitioned table. PostHog created a partitioned Persons table with a fresh TOAST table and OID space, kept old and new tables in sync with database triggers, copied existing Person data with a backfill script, and changed ingestion and application code so that all writes went to the new table and reads tried it first, falling back to the old one. The web app was deployed under manual control, and engineers worked in shifts through the weekend on the feature flags API, error tracking ingestion and the Django web application. Ingestion was healthy again on 14 November at 15:02 UTC.

While ingestion workers were scaled up to clear the backlog, local disk on the AWS MSK cluster reached about 85%, so PostHog paused ingestion, reduced the topic's local retention and resumed once there was room. The backfill was moved into Dagster for better visibility. By 15 November at 6:20 AM UTC all events since the start had been processed. No data was lost.

Severity
–
Scope
Event and persons ingestion for US Cloud, delayed from 10 minutes up to 2 days; analytics on recent data, feature flags and experiments using person properties, filtering in error tracking and session replay, and CDP destinations and workflows were affected
Users affected
All customers in the US region
Region
US Cloud
Time to detect
–
Time to resolve
3d 14h 18m

Symptoms

From 11 November 2025 at 16:02 UTC, PostHog's event ingestion pipeline for US Cloud became severely degraded, and events for all customers in the US region were processed with delays from 10 minutes up to 2 days. Writes to the Postgres database that stores Person information could take multiple seconds per update, and the pipeline slowed until no standard scaling option helped.

Recent events did not show up in the product. Charts and queries over recent time ranges were inaccurate, feature flags that relied on person properties evaluated with delayed properties, filtering of errors and session replays by Person updates was affected, and CDP destinations and workflows that relied on Persons ran late. Ingestion of errors and replays itself stayed healthy.

Standard dashboards for CPU, memory, IOPS and lock contention did not point at the cause, and there was no monitoring of TOAST table size, OID usage or disk reads per write on a table. The root cause was found only on 12 November at 10:17pm UTC, a day and a half after the start, with help from AWS RDS engineers and external consultants. Disk reads per write query had grown from 10kb to 15MB, and the latency of those queries had risen 100x.

During the catch-up, local disk usage on the AWS MSK (Kafka) cluster reached about 85% and triggered alerts, because tiered storage keeps the most recent 4 hours of data on local disk before moving it to S3, and the backlog made that window unusually dense.

Symptom Throughput degradationHigh latency

Root cause

Root cause Capacity exhaustion

Component DatabaseMessage broker
Contributing factor Unknown limitMonitoring blind spotNo runbook
Trigger
The 32-bit OID space used to track TOASTed values of the Persons table's properties column, a JSONb field that often exceeds the 2KB threshold for out-of-line storage, was nearly exhausted.

Detection

–

Remediation

Action items

Lessons

TOAST OID exhaustion degrades writes long before the hard limit of about 4 billion OIDs, it is rare even at large scale, and standard dashboards for CPU, memory, IOPS and lock contention did not immediately point at it, so diagnosis took a day and a half without monitoring of TOAST growth, OID usage or disk reads per write. Many core features depended on timely updates to the persons table, so its slowdown reached a wide part of the product. The first backfill approach lacked visibility for a long migration, the Kafka tiered storage settings and alerts were not tuned for a large backlog, and the people who normally handle customer communications were all unavailable.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Message broker among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.