Incident Library

IL-0286 Honeycomb · 2025-12-05databaseawsobservability

An exercise cleanup destroyed Honeycomb's Kafka brokers, leaving partitions leaderless and ingest crash-looping

Honeycomb runs disaster recovery exercises every year, in production too. Each Honeycomb instance runs in one region over at least three availability zones, and in production each Kafka topic partition is replicated to one broker per zone. When a zone fails, an AWS auto-scaling group boots replacement brokers in working zones, which then copy data from the in-sync replicas. In the cleanup after the 5 December exercise, the brokers that were killed included older in-sync ones, so some partitions were left with no broker at all. The EU pre-production clusters have fewer nodes: there one broker is killed in the test and one in the cleanup, which makes it far less likely that a partition loses all its brokers, so four pre-production runs had not shown the flaw. Retriever stores the last Kafka offset it has seen and refuses to process data when offsets reset, so a dirty leader election could break Retriever ingestion for good. Kafka here also uses Confluent tiered storage, which keeps fresh data on local disks and older data on S3.

About an hour in, responders had found the affected customers. The impacted partitions were marked read-only on the producer side, which stopped Shepherd's crash loop, and reassignments brought customers back. The Kafka autobalancer was turned off and a manual reassignment ran overnight, local retention was lowered around midnight PT, and on Saturday, with disk running out, responders planned to shut ingest at the ALB at 95% usage, cancelled the main reassignments and repaired the broken _confluent-tier-state and __consumer_offsets partitions, which also unblocked SLO processing. Ingest was turned off at 7 a.m. PT, and while new brokers, Retrievers and partitions were being prepared, tiered storage was turned off, disk space came back at once and ingest was reopened, ending the customer impact. Fully damaged Retriever partitions were truncated and left read-only, so older data could not be queried for a while but was not lost.

In the second week Honeycomb extended the binlog retention of its MySQL read-only replica, because Activity Log could no longer publish events to Kafka, and patched Retriever so old data could be queried despite broken offsets. An attempt to turn tiered storage back on failed. On Wednesday a topic delete timed out and left the cluster unable to do any administrative work beyond listing topics, so the team split into one lane trying low-risk fixes and ZooKeeper surgery and one planning a full evacuation to a new Kafka cluster, with five scenarios depending on what would be ready. On Friday the Storage team made a one-time offset reset work for Retriever, and the platform team had made good progress on feature flags to switch readers and writers and had made it possible to boot a second Kafka cluster. After a dress rehearsal in an EU pre-production cluster on Monday, which took four hours, and a fix for NLB domain names that did not fit AWS limits, the migration on Tuesday took two hours with at most one hour of delayed signal. An attempt to stretch MySQL binlog retention with a stalled replica did not work, and Activity Log data from 5 December at ~6 p.m. ET to 9 December at ~6 p.m. ET was lost.

Severity
–
Scope
Ingest failed for teams and environments whose partitions were all unavailable; one Retriever partition, about 2.5% of the cluster, lost all its data; older data on truncated partitions could not be queried for a while; SLO processing saw ~18h of late data; Activity Log data from 5 December ~6 p.m. ET to 9 December ~6 p.m. ET was lost
Users affected
the Honeycomb teams and environments assigned to the damaged partitions
Region
–
Time to detect
–
Time to resolve
–

Symptoms

On 5 December, during the cleanup after a zonal failover exercise in production, Honeycomb destroyed Kafka brokers on purpose as a runbook step, and several topic partitions lost every broker that held them and became fully leaderless. Any Honeycomb team or environment whose assigned partitions were all unavailable could no longer have its data ingested.

The damage reached three kinds of topics: the topics that carry all ingested events, some of whose partitions were damaged beyond repair; the topics some consumer groups use to track their position; and metadata topics of Kafka and of Confluent add-ons such as tiered storage, some with fully broken partitions. The ingest service Shepherd had gone into a crash loop more than an hour before responders noticed it through the alerting noise, because it buffered in memory the events meant for the broken partitions. About a third of the partitions used by Retriever, Honeycomb's query engine, were affected, and one of them, about 2.5% of the cluster, lost all its data.

The Kafka cluster then got steadily worse. Overnight reassignments made no visible progress while broker disk usage kept rising; on Saturday morning the brokers were losing 5% of disk capacity per hour with 30% free, and at 7 a.m. PT, with 5.07% free, ingest was turned off, yet disk usage only stabilised instead of falling. When two broken metadata topics were repaired, SLO triggers fired about ~18h of late data, because the SLO processing service had not seen any new data in that time. On the following Wednesday a topic delete operation timed out, and after that no Kafka broker could do anything administrative except list topics: describing, reassigning or modifying topics timed out.

Responders could not run an ordinary leader election, because a "dirty" leader election after losing all brokers of a partition can roll back Kafka offsets, and Retriever refuses to process data when offsets go back. Turning off the Kafka autobalancer, reassigning partitions by hand, lowering local retention and moving smaller partitions did not free disk space. Turning off tiered storage, which an engineer suggested because it had worked in an outage in 2021, restored the disk space at once. Later attempts at dirty leader elections for the Activity Log partitions failed.

Symptom Container crash loopData lossNode disk fullMonitoring outage

Root cause

Root cause Operator error

Component Message broker
Contributing factor No safeguard on destructive changeUnrepresentative stagingSignal buried in noiseHarmful manual action
Trigger
During the cleanup that rebalances workloads after a successful zonal failover exercise, Honeycomb followed its runbook and destroyed Kafka brokers on purpose. Older, in-sync brokers were killed by accident, so some topic partitions lost every broker that held them.

Detection

Responders noticed through the alerting noise that the ingest service Shepherd had gone into a crash loop more than an hour earlier. On Saturday morning they saw that the Kafka balancing jobs were not progressing and disk usage had not improved, and once the metadata topics were repaired, SLO processing triggers fired about ~18h of late data.

Remediation

Action items

Lessons

Many elements of this outage were already known: similar outages, workaround plans, RFCs, migrations and procedures for running such exercises safely existed, but the documents were unknown outside their authors or hard to find. Flexible, locally controlled work is easy to defer again and again in favour of urgent work, and Honeycomb wants to notice when deferring becomes a reflex. Many key interventions came from people who no longer owned the components, which shows drift between team boundaries and the coupling inside the system. The pre-production clusters were too small to reproduce the flaw, and tests on a cluster closer to production would have made catching it much more likely. Coordination and public communication were also hard: senior engineers tended to take the lead automatically, and customers said the updates gave too much detail about minor beta features.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Message broker among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.