Cases2 cases

1 to 2 of 2

1. IL-0390 PostHog US Cloud event ingestion slowed to delays of up to 2 days as writes to the Persons Postgres table stalled

   PostHog2025

   Symptoms

   From 11 November 2025 at 16:02 UTC, PostHog's event ingestion pipeline for US Cloud became severely degraded, and events for all customers in the US region were processed with delays from 10 minutes up to 2 days. Writes to the Postgres database that stores Person information could take multiple seconds per update, and the pipeline slowed until no standard scaling option helped.

   Throughput degradationHigh latency

   Trigger

   The 32-bit OID space used to track TOASTed values of the Persons table's properties column, a JSONb field that often exceeds the 2KB threshold for out-of-line storage, was nearly exhausted.

   Root cause

   Capacity exhaustion

   Component

   DatabaseMessage broker

   Contributing factor

   Unknown limitMonitoring blind spotNo runbook

   Remediation

   • Created a new partitioned Persons table with a fresh TOAST table and OID space
   • Added database triggers to keep the old and new Persons tables in sync for live writes
   • Copied the existing Person data from the old table to the new one with a backfill script
   • Changed ingestion and application code to write to the new table and read from it first, with fallback to the old table
   • Deployed the web app changes to production under manual control rather than a fully automated rollout
   • Scaled ingestion workers to process the accumulated backlog
   • Paused ingestion when the MSK local disk reached about 85%
   • Reduced the local retention of the affected Kafka topic
   • Resumed ingestion after confirming there was enough disk headroom
   • Moved the backfill into Dagster to finish it over the weekend

   Action items

   • Add metrics and alerts for heavy disk read amplification per write
   • Add metrics and alerts for TOAST table statistics
   • Add metrics and alerts for other Postgres engine limits that matter at large scale
   • Write runbooks for engine-level Postgres limits, with decision trees for migrating versus repairing in place
   • Create customer communication runbooks with a clear escalation path and redundancies
   • Evaluate other data stores for the persons database
   • Complete the reconsolidation on the new Persons tables
   • Improve the backfill tooling and the safeguards for Kafka tiered storage
   • Design and operate Person-like tables so they avoid similar limits

   Lessons

   TOAST OID exhaustion degrades writes long before the hard limit of about 4 billion OIDs, it is rare even at large scale, and standard dashboards for CPU, memory, IOPS and lock contention did not immediately point at it, so diagnosis took a day and a half without monitoring of TOAST growth, OID usage or disk reads per write. Many core features depended on timely updates to the persons table, so its slowdown reached a wide part of the product. The first backfill approach lacked visibility for a long migration, the Kafka tiered storage settings and alerts were not tuned for a large backlog, and the people who normally handle customer communications were all unavailable.

   View case
2. IL-0275 Parse.ly's Kafka brokers hit network limits and failed one after another, stopping the analytics pipeline for hours

   Parse.ly2015

   Symptoms

   On Thursday night, 26 March 2015, and again on Friday, 27 March, Parse.ly's data processing backend stopped. Monitoring tools such as Munin, Sentry and Pingdom reported failures widely, PagerDuty woke the on-call engineer, and no new analytics data appeared in the dashboards at dash.parsely.com and preview.parsely.com for several hours.

   Service outageDNS resolution failures

   Trigger

   Data volume reached a point where the Kafka brokers, running on modest Amazon EC2 nodes without high network capacity, hit operating system network limits under the outbound read traffic and became unavailable.

   Root cause

   Capacity exhaustion

   Component

   Message brokerService discovery

   Contributing factor

   Monitoring blind spotUnknown limitNo headroomHarmful manual action

   Detection

   Munin, Sentry and Pingdom reported the failures widely, Graphite graphs showed them, and PagerDuty woke the on-call engineer when the first outages started in the night.

   Remediation

   • Added new brokers to expand the Kafka cluster
   • Corrected the configuration of the new brokers and deployed them to the cluster, after which network utilization per node dropped and processing resumed
   • Turned the downstream consumer systems back on slowly through Apache Storm and reprocessed the backlog
   • Began restoring the data to customer accounts from the raw data archived in Amazon S3

   Lessons

   CPU and disk I/O are good signs of machine health, but creeping network utilization is harder to monitor, because 100% of network capacity is not well defined on cloud VMs that share host machines. When every node of a cluster runs close to the same limit, losing one node can take down the whole cluster, so failure planning should cover an entire subsystem going offline, not just one or two machines. Keeping raw data archival in simple files and S3 uploads, separate from the Kafka stream, meant there was no risk of permanent data loss.

   View case

By root cause1