Cases4 cases

1 to 4 of 4

1. IL-0281 PostHog's Logs ClickHouse cluster silently deleted almost all its S3 data, losing US logs older than 3 days

   PostHog2026

   Symptoms

   On 19 February 2026, a node of the ClickHouse cluster behind PostHog's Logs product crashed, restarted and could not start again because it could not find a large amount of its data. Engineers then found that almost all of the Logs data stored on S3 was gone, so logs collected in the US region more than 3 days earlier were lost. Only the Logs product was affected.

   Data loss

   Trigger

   At 10:54 UTC a routine mutation was run to materialise an index. When it finished at 11:02, it set off a bug in ClickHouse's zero-copy replication.

   Root cause

   Software defect

   Component

   DatabaseDatabase replicationObject storageDatabase backups and snapshots

   Contributing factor

   Monitoring blind spotSingle point of failureUntested backup

   Detection

   No automated alarm fired. At 19:40 UTC a node in the cluster crashed, restarted and failed to start because of the missing data, and the engineers who investigated found that most of the S3-backed data was gone.

   Remediation

   • Switched Logs to a new table without zero-copy replication and caught up on recent messages
   • Backfilled the Logs data from the 3 days of Kafka history

   Action items

   • Disable zero-copy replication on all clusters
   • Audit the whole Logs ClickHouse configuration and keep experimental features out of production
   • Enable S3 versioning on the storage buckets so objects deleted by a database bug can be recovered
   • Spot-check configurations and data integrity strategies for single points of failure before a product becomes generally available

   Lessons

   Running Logs in its own cluster kept every other product unaffected, and 3 days of Kafka retention saved the recent data. An experimental setting chosen for an early internal stage was allowed to stay in production. Monitoring covered only incoming and recent data, so petabytes were deleted over 8 hours without an alarm. Relying on database replication alone over shared storage was a single point of failure, and S3 versioning would have let the deleted files be recovered.

   View case
2. IL-0378 All Cloudflare R2 object writes and about 35% of reads failed when the R2 Gateway could not authenticate to storage

   Cloudflare2025

   Symptoms

   From 21:38 UTC on 21 March 2025, Cloudflare R2 object storage returned errors for 1 hour and 7 minutes: all object writes failed and about 35% of object reads failed globally. R2 global availability alerts fired at 21:45 UTC, at a 2% error budget burn rate.

   Credentials unavailableHigh error rate

   Trigger

   During a routine rotation of the credentials the R2 Gateway Worker uses to authenticate with the storage infrastructure, the new credentials were deployed to the Worker of the default environment instead of the production Worker, and the old credentials were then removed from the storage infrastructure at 20:37 UTC.

   Root cause

   Operator error

   Component

   Object storage

   Contributing factor

   Unverified assumptionMonitoring blind spotChange with a delayed effect

   Detection

   R2 global availability alerts fired at 21:45 UTC, seven minutes after impact began, at a 2% error budget burn rate, and the R2 engineering team turned to operational dashboards and logs.

   Remediation

   • Deployed the new credentials to the production R2 Gateway Worker, which restored R2 availability

   Action items

   • Add logging tags with the suffix of the credential ID the R2 Gateway Worker uses
   • Require confirmation that the new token ID suffix matches storage infrastructure logs before deleting the old token
   • Run key rotation through the hotfix release tooling, which enforces the environment, instead of manual wrangler commands
   • Write the two-person validation of these changes into the standard operating procedures
   • Extend the closed loop health check system to test new keys, report their status through alerting and confirm global propagation before releasing the gateway Worker
   • Add observability views of upstream success rates that bypass caching

   Lessons

   Because the deletion of the old credentials took effect gradually, availability metrics were a poor way to confirm the change. Cloudflare said the team should have checked directly which token the R2 Gateway service was using to authenticate with the storage infrastructure instead of relying on those metrics.

   View case
3. IL-0343 DNS names under digitaloceanspaces.com stopped resolving, breaking Spaces, Container Registry and the Control Panel

   DigitalOcean2024

   Symptoms

   From 22:38 UTC on 28 November 2024, DigitalOcean's Cloud Control Panel stopped loading for customers, and sign-up and log-in failed. DigitalOcean declared the incident from monitoring and customer reports, an internal alert fired at 22:44 UTC, and Spaces buckets and the services that depend on them began to fail intermittently.

   DNS resolution failuresHigh error rateImage pull failures

   Trigger

   At 22:38 UTC on 28 November 2024, DigitalOcean's domain registrar, Network Solutions, made an update to the digitaloceanspaces.com domain that added an Extensible Provisioning Protocol (EPP) clientHold status code to it.

   Root cause

   Dependency failure

   Component

   Domain registrar and registryObject storageContainer registry and its credentials

   Contributing factor

   Cache-masked failureMonitoring blind spot

   Detection

   DigitalOcean declared the incident at 22:38 UTC based on monitoring and customer reports that the Cloud Control Panel would not load. An internal alert detected the issue at 22:44 UTC, and the first customer contact about the Control Panel came at 22:57 UTC.

   Remediation

   • Deployed a change that brought the Cloud Control Panel back online
   • Contacted Network Solutions to have the clientHold removed
   • Escalated to Verisign executives, the .com registry, until the clientHold was removed

   Action items

   • Work with Network Solutions to understand why the clientHold was applied to the domain
   • Review other domain registrars as possible new homes for DigitalOcean's domains
   • Review monitoring and alerting to detect registrar-imposed changes and DNS resolution problems sooner

   View case
4. IL-0381 A misbehaving back-end node in Linode's Newark Object Storage slowed the service and caused 504 gateway timeouts

   Linode2022

   Symptoms

   In the week of 10 October 2022, Linode Object Storage in Newark had two incidents of reduced performance and availability, on 10 and 16 October, and in some cases the front-end gateways returned 504 gateway timeouts to customers. The second incident began while the first was still being investigated.

   Intermittent timeoutsThroughput degradation

   Trigger

   One back-end Object Storage node had a hardware problem that was not an outright failure, so the node kept going up and down.

   Root cause

   Hardware failure

   Component

   Object storage

   Contributing factor

   Monitoring blind spotAutomation amplification

   Remediation

   • Removed the failed back-end infrastructure from the Object Storage cluster

   Action items

   • Improve detection so subtle problems with failing or misbehaving hardware are found and triaged before they affect the wider cluster
   • Describe Ceph-related events more accurately to on-call administrators and define alerts more clearly

   Lessons

   Hardware that misbehaves without failing outright can cause service-wide problems, and Linode's internal monitoring had gaps in spotting such subtle issues before they affected the wider cluster.

   View case

By root cause4