Cases1 case

1 to 1 of 1

1. IL-0286 An exercise cleanup destroyed Honeycomb's Kafka brokers, leaving partitions leaderless and ingest crash-looping

   Honeycomb2025

   Symptoms

   On 5 December, during the cleanup after a zonal failover exercise in production, Honeycomb destroyed Kafka brokers on purpose as a runbook step, and several topic partitions lost every broker that held them and became fully leaderless. Any Honeycomb team or environment whose assigned partitions were all unavailable could no longer have its data ingested.

   Container crash loopData lossNode disk fullMonitoring outage

   Trigger

   During the cleanup that rebalances workloads after a successful zonal failover exercise, Honeycomb followed its runbook and destroyed Kafka brokers on purpose. Older, in-sync brokers were killed by accident, so some topic partitions lost every broker that held them.

   Root cause

   Operator error

   Component

   Message broker

   Contributing factor

   No safeguard on destructive changeUnrepresentative stagingSignal buried in noiseHarmful manual action

   Detection

   Responders noticed through the alerting noise that the ingest service Shepherd had gone into a crash loop more than an hour earlier. On Saturday morning they saw that the Kafka balancing jobs were not progressing and disk usage had not improved, and once the metadata topics were repaired, SLO processing triggers fired about ~18h of late data.

   Remediation

   • Marked the impacted partitions read-only on the producer side, which stopped Shepherd's crash loop
   • Ran partition reassignments to bring customers back to normal
   • Turned off the Kafka autobalancer and left a manual reassignment plan running overnight
   • Reduced local retention on Kafka to rely on S3 tiered storage for longer retention
   • Repaired the broken _confluent-tier-state and __consumer_offsets partitions, which unblocked SLO processing
   • Turned off ingest to stop the brokers from running out of disk
   • Turned off tiered storage, which freed the broker disk space at once
   • Reopened ingest, which ended the customer impact
   • Truncated the fully damaged Retriever partitions and left them read-only
   • Extended the binlog retention of the MySQL read-only replica because Activity Log could not publish to Kafka
   • Patched Retriever so its nodes could idle live and serve old data despite broken offsets
   • Made a one-time offset reset work in Retriever
   • Evacuated to a new Kafka cluster with new topics and partitions
   • Booted an Activity Log database instance from 9 December and replicated from it to limit data loss
   • Cleaned up the older Kafka clusters

   Action items

   • Consolidate internal documentation better as the company grows

   Lessons

   Many elements of this outage were already known: similar outages, workaround plans, RFCs, migrations and procedures for running such exercises safely existed, but the documents were unknown outside their authors or hard to find. Flexible, locally controlled work is easy to defer again and again in favour of urgent work, and Honeycomb wants to notice when deferring becomes a reflex. Many key interventions came from people who no longer owned the components, which shows drift between team boundaries and the coupling inside the system. The pre-production clusters were too small to reproduce the flaw, and tests on a cluster closer to production would have made catching it much more likely. Coordination and public communication were also hard: senior engineers tended to take the lead automatically, and customers said the updates gave too much detail about minor beta features.

   View case

By root cause1