Incident Library

IL-0281 PostHog · 2026-02-19databaseaws

PostHog's Logs ClickHouse cluster silently deleted almost all its S3 data, losing US logs older than 3 days

PostHog stores Logs data in a dedicated ClickHouse cluster, separate from the main ClickHouse cluster shared by most other PostHog products. The Logs cluster uses S3 disks: data parts are uploaded to S3 automatically after 24 hours, which lets it hold very large volumes (PostHog's own systems alone produce about 500MB/s of logs, or about 1PB/month uncompressed). The database has two replicas, but zero-copy replication, an experimental ClickHouse feature that ClickHouse does not recommend for production, had been turned on very early, when Logs was an internal-only tool, and the setting was not revisited when Logs was released to external users. With zero-copy replication the replicas share the same S3 objects, so there was no redundancy between the database logic and the storage, and S3 versioning was not enabled on the bucket.

At 11:02 UTC on 19 February the finished mutation triggered a zero-copy replication bug, and one replica wrongly concluded that none of the data parts in the database were referenced any more. Until 19:40 the replicas deleted the S3 data of the whole database; a fault that should have removed one replica's data removed it everywhere. At 19:40 a node crashed and could not start, and engineers found most S3-backed data missing. At 21:45 the data was judged most likely unrecoverable and disaster recovery began. At 22:15 the team decided to move to a new table and restore data from Kafka, which kept 3 days of retention, and by 23:00 the new table, without zero-copy replication, was in use and had caught up on recent messages. The backfill from Kafka history ran on 20 February from 10:05 to 12:21 UTC. Data older than the Kafka retention could not be restored.

Severity
–
Scope
Logs product: logs collected more than 3 days before the incident were lost; all other PostHog products and data were unaffected
Users affected
–
Region
US region
Time to detect
8h 38m
Time to resolve
25h 19m

Symptoms

On 19 February 2026, a node of the ClickHouse cluster behind PostHog's Logs product crashed, restarted and could not start again because it could not find a large amount of its data. Engineers then found that almost all of the Logs data stored on S3 was gone, so logs collected in the US region more than 3 days earlier were lost. Only the Logs product was affected.

The deletion had run in the background for about eight hours, from 11:02 to 19:40 UTC, without a single alarm. The Logs cluster moves data parts to S3 only after 24 hours and most queries read recent logs, so query errors stayed few. Alerts covered ingestion lag, query latency and error rates on active queries, nothing measured the integrity of the older data on S3, and the deletion raised no application-level errors, so dashboards stayed green until the node restart forced a consistency check.

Symptom Data loss

Root cause

Root cause Software defect

Component DatabaseDatabase replicationObject storageDatabase backups and snapshots
Contributing factor Monitoring blind spotSingle point of failureUntested backup
Trigger
At 10:54 UTC a routine mutation was run to materialise an index. When it finished at 11:02, it set off a bug in ClickHouse's zero-copy replication.

Detection

No automated alarm fired. At 19:40 UTC a node in the cluster crashed, restarted and failed to start because of the missing data, and the engineers who investigated found that most of the S3-backed data was gone.

Remediation

Action items

Lessons

Running Logs in its own cluster kept every other product unaffected, and 3 days of Kafka retention saved the recent data. An experimental setting chosen for an early internal stage was allowed to stay in production. Monitoring covered only incoming and recent data, so petabytes were deleted over 8 hours without an alarm. Relying on database replication alone over shared storage was a single point of failure, and S3 versioning would have let the deleted files be recovered.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Database replication, Object storage among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.