Graph
Cases4 cases
1 to 4 of 4
IL-0286An exercise cleanup destroyed Honeycomb's Kafka brokers, leaving partitions leaderless and ingest crash-loopingHoneycomb2025- Symptoms
On 5 December, during the cleanup after a zonal failover exercise in production, Honeycomb destroyed Kafka brokers on purpose as a runbook step, and several topic partitions lost every broker that held them and became fully leaderless. Any Honeycomb team or environment whose assigned partitions were all unavailable could no longer have its data ingested.
- Trigger
During the cleanup that rebalances workloads after a successful zonal failover exercise, Honeycomb followed its runbook and destroyed Kafka brokers on purpose. Older, in-sync brokers were killed by accident, so some topic partitions lost every broker that held them.
- Root cause
- Component
- Contributing factor
- Detection
Responders noticed through the alerting noise that the ingest service Shepherd had gone into a crash loop more than an hour earlier. On Saturday morning they saw that the Kafka balancing jobs were not progressing and disk usage had not improved, and once the metadata topics were repaired, SLO processing triggers fired about ~18h of late data.
- Remediation
- Marked the impacted partitions read-only on the producer side, which stopped Shepherd's crash loop
- Ran partition reassignments to bring customers back to normal
- Turned off the Kafka autobalancer and left a manual reassignment plan running overnight
- Reduced local retention on Kafka to rely on S3 tiered storage for longer retention
- Repaired the broken _confluent-tier-state and __consumer_offsets partitions, which unblocked SLO processing
- Turned off ingest to stop the brokers from running out of disk
- Turned off tiered storage, which freed the broker disk space at once
- Reopened ingest, which ended the customer impact
- Truncated the fully damaged Retriever partitions and left them read-only
- Extended the binlog retention of the MySQL read-only replica because Activity Log could not publish to Kafka
- Patched Retriever so its nodes could idle live and serve old data despite broken offsets
- Made a one-time offset reset work in Retriever
- Evacuated to a new Kafka cluster with new topics and partitions
- Booted an Activity Log database instance from 9 December and replicated from it to limit data loss
- Cleaned up the older Kafka clusters
- Action items
- Consolidate internal documentation better as the company grows
- Lessons
Many elements of this outage were already known: similar outages, workaround plans, RFCs, migrations and procedures for running such exercises safely existed, but the documents were unknown outside their authors or hard to find. Flexible, locally controlled work is easy to defer again and again in favour of urgent work, and Honeycomb wants to notice when deferring becomes a reflex. Many key interventions came from people who no longer owned the components, which shows drift between team boundaries and the coupling inside the system. The pre-production clusters were too small to reproduce the flaw, and tests on a cluster closer to production would have made catching it much more likely. Coordination and public communication were also hard: senior engineers tended to take the lead automatically, and customers said the updates gave too much detail about minor beta features.
- Symptoms
On 2025-03-12 from 10:58 UTC, alerts for high backend response times came in, and the MediaWiki deployments mw-api-int, mw-parsoid and mw-jobrunner, which serve MediaWiki for non-user-facing purposes, showed high latency and database errors. The External Storage cluster, whose hosts store the compressed text of wiki page revisions, had a sharp rise in load and connections. The same happened again on 2025-03-13 from 11:15 UTC.
- Trigger
At 10:44 UTC on 12 March a Scap deployment of the PHP 8.1 rollout for mw-api-int, mw-parsoid and mw-jobrunner was run with the flag -Dbuild_mw_container_image:False; the rollout was deployed with Scap again at 11:15 UTC on 13 March.
- Root cause
- Component
- Contributing factor
- Detection
Alerts for high backend response times started at 10:58 UTC on 12 March, 14 minutes after the Scap deployment; 2 on-call engineers were paged. The MediaWikiMemcachedHighErrorRate alert that pointed at the cause fired only briefly because its log pipeline was overloaded.
- Remediation
- Deployed a patch to cirrus-streaming-updater that reduced SUP parallelism
- Reduced the concurrency of the categoryMembershipChange job in changeprop-jobqueue
- On 12 March, reverted 1126607 and 1126650 and re-ran Scap
- On 13 March, rolling-restarted changeprop-jobqueue
- Rolling-restarted mw-api-int
- Disabled the categorymembership job in changeprop-jobqueue
- Reverted 1127476 and 1127478
- Redeployed mw-api-int to pick up the revert
- Increased the number of mw-api-int replicas
- Redeployed mw-jobrunner to pick up the revert
- Redeployed mw-parsoid to pick up the revert
- Action items
- Reduce the number of messages sent through channel:Memcached during failures
- Add helm rollback functionality to Scap
- Decide whether Scap should update helmfile-defaults when -Dbuild_mw_container_image:False is set
- Consider removing envvars.inc from MediaWiki images
- Consider "Why" dashboards that give an overview of a group of related components together with relevant links to follow
- Coordinate incident responders better
- Be able to work with MediaWiki developers during incidents
- Lessons
The write-up puts the weight not on the technical root cause but on why the team could not see, even after the 12 March incident, that three MediaWiki deployments were running without Memcached. Alert dashboards showed what was happening, such as many busy PHP workers, but not why; dashboards linking components that affect each other, such as SQLBlobStore keys and External Storage, would have helped. The alert that pointed in the right direction did not stay firing and would probably have been lost among 78 others. The incident coordinator could not tell who was working on what or who had joined or left, the channels were noisy, the external impact was judged more critical than it was, and trace.wikimedia.org was not used although it might have helped.
IL-0280Kafka brokers in many New Relic cells ran out of disk, leaving US data collection and alerting unusable for hoursNew Relic2021- Symptoms
On 29 July 2021, data collection and alerting in New Relic's US region became unusable for many customers for several hours. It began with one unresponsive Kafka broker in a single cell of NRDB, New Relic's telemetry data platform, and over the following hours Kafka brokers in more and more cells failed as their disks filled up.
- Trigger
Responders used an incident response tool to extend Kafka data retention for the degraded cell so that its backlog would not be lost. The retention increase was larger than usual, and the tool applied it to all cells of the same type, not only the affected one.
- Root cause
- Component
- Contributing factor
- Detection
Engineers saw a Kafka broker in one cell become unresponsive on the morning of 29 July. Later a monitoring alert warned that disk space was running low, but it fired together with unrelated warnings from several other systems and was missed.
- Remediation
- Restarted the unresponsive Kafka broker by hand, while the automated restart was also running
- Resolved the disk space problem on the Kafka brokers, after which service recovered slowly
- Lessons
Cell isolation is the strongest protection against a widespread disruption, and emergency tools have to respect it just as normal operations do. Emergency response tools must be safe to use in chaotic situations without losing their ability to work. Engineers' attention is limited, so incident response processes and alerts need regular review to direct it at the most important signal.
- Symptoms
On 15 December 2016 from 1.17am, VMs of the GOV.UK Platform as a Service (PaaS) in AWS eu-west-1 logged DNS resolution errors, and applications deployed on the PaaS failed DNS queries intermittently. Applications could fail to connect to their databases, backend services and APIs, and some end users of those applications would have seen intermittent slowness and errors between 1am and 3.17am.
- Root cause
- Component
- Contributing factor
- Detection
Datadog and Pingdom alerts fired during the night, but no one was on out-of-hours support. A PaaS team member noticed the warnings at 8am when starting work and checked monitoring to confirm the issue was over.
- Action items
- Stop the monitoring system from sending emails when disk monitoring data is not being received
- Make notifying tenants about outages easier and quicker
- Implement alert notifications for out-of-hours support, shaped by this incident
- Provide 24/7 support for the PaaS in line with the needs of adopting services
These 4 cases involve Message broker and In-memory cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.
By root cause3
Operator error2 cases
- Component
- Contributing factor
- Organisation
Faulty deployment1 case
- Contributing factor
- Organisation
Cloud provider outage1 case
- Component
- Contributing factor
- Organisation