Cases5 cases

1 to 5 of 5

1. IL-0286 An exercise cleanup destroyed Honeycomb's Kafka brokers, leaving partitions leaderless and ingest crash-looping

   Honeycomb2025

   Symptoms

   On 5 December, during the cleanup after a zonal failover exercise in production, Honeycomb destroyed Kafka brokers on purpose as a runbook step, and several topic partitions lost every broker that held them and became fully leaderless. Any Honeycomb team or environment whose assigned partitions were all unavailable could no longer have its data ingested.

   Container crash loopData lossNode disk fullMonitoring outage

   Trigger

   During the cleanup that rebalances workloads after a successful zonal failover exercise, Honeycomb followed its runbook and destroyed Kafka brokers on purpose. Older, in-sync brokers were killed by accident, so some topic partitions lost every broker that held them.

   Root cause

   Operator error

   Component

   Message broker

   Contributing factor

   No safeguard on destructive changeUnrepresentative stagingSignal buried in noiseHarmful manual action

   Detection

   Responders noticed through the alerting noise that the ingest service Shepherd had gone into a crash loop more than an hour earlier. On Saturday morning they saw that the Kafka balancing jobs were not progressing and disk usage had not improved, and once the metadata topics were repaired, SLO processing triggers fired about ~18h of late data.

   Remediation

   • Marked the impacted partitions read-only on the producer side, which stopped Shepherd's crash loop
   • Ran partition reassignments to bring customers back to normal
   • Turned off the Kafka autobalancer and left a manual reassignment plan running overnight
   • Reduced local retention on Kafka to rely on S3 tiered storage for longer retention
   • Repaired the broken _confluent-tier-state and __consumer_offsets partitions, which unblocked SLO processing
   • Turned off ingest to stop the brokers from running out of disk
   • Turned off tiered storage, which freed the broker disk space at once
   • Reopened ingest, which ended the customer impact
   • Truncated the fully damaged Retriever partitions and left them read-only
   • Extended the binlog retention of the MySQL read-only replica because Activity Log could not publish to Kafka
   • Patched Retriever so its nodes could idle live and serve old data despite broken offsets
   • Made a one-time offset reset work in Retriever
   • Evacuated to a new Kafka cluster with new topics and partitions
   • Booted an Activity Log database instance from 9 December and replicated from it to limit data loss
   • Cleaned up the older Kafka clusters

   Action items

   • Consolidate internal documentation better as the company grows

   Lessons

   Many elements of this outage were already known: similar outages, workaround plans, RFCs, migrations and procedures for running such exercises safely existed, but the documents were unknown outside their authors or hard to find. Flexible, locally controlled work is easy to defer again and again in favour of urgent work, and Honeycomb wants to notice when deferring becomes a reflex. Many key interventions came from people who no longer owned the components, which shows drift between team boundaries and the coupling inside the system. The pre-production clusters were too small to reproduce the flaw, and tests on a cluster closer to production would have made catching it much more likely. Coordination and public communication were also hard: senior engineers tended to take the lead automatically, and customers said the updates gave too much detail about minor beta features.

   View case
2. IL-0242 Both PostgreSQL servers behind matrix.org were lost during a disk expansion, and the homeserver was down for 24 hours

   Matrix.org Foundation2025

   Symptoms

   On 2 September 2025 the matrix.org homeserver went down at 17:45 UTC and stayed down until 18:00 UTC the next day. Both machines of its PostgreSQL cluster were out of action: the original primary had lost its RAID array during a disk expansion, and the data directory of the new primary had been deleted by mistake.

   Service outageData loss

   Trigger

   During maintenance to add drives, an existing drive disappeared from the primary database server's RAID 10 array at 11:17 UTC, which led to a failover and a rebuild of that server. The outage itself began when the data directory of the new primary on db-01 was deleted while clearing a failed restore on db-02.

   Root cause

   Operator error

   Component

   DatabaseDatabase backups and snapshotsDatabase replication

   Contributing factor

   Overlapping failuresHarmful manual actionUnpatched known bugNo safeguard on destructive change

   Detection

   Monitoring fired when the drive disappeared from db-02's RAID array at 11:17 UTC, and Mythic Beasts confirmed the issue. The team realised it had deleted db-01's data directory, and the backend team was then paged, an incident was opened and an emergency was declared.

   Remediation

   • Created a fresh RAID 10 array on db-02
   • Unblocked WAL archiving from the new primary on db-01 to S3
   • Started restoring db-02 from the S3 backups with wal-g
   • Shut down Synapse and the damaged primary database
   • Remounted the data partition of db-01 read-only
   • Copied the WALs missing from S3 from db-01 to db-02
   • Updated wal-g and patched it to continue from the partial restore
   • Configured a restore_command wrapper that fetched WALs from S3 and the local files, and replayed them in standby mode
   • Promoted db-02 to primary and restarted Synapse
   • Ran backups against the primary while no replica existed
   • Rebuilt db-01 as a replica from a base backup of db-02

   Action items

   • Run pg_repack to reclaim free space
   • Take incremental backups more often, since they restore faster than WAL replay
   • Reduce Synapse's storage requirements
   • Consider changing the terminal background colour by the database's role, for example by watching for the standby.signal file
   • Consider wrapper scripts around sensitive commands, such as an alias for rm, and automating operations such as starting a base backup from primary to secondary
   • Shorten the recovery time, with infrastructure from service providers
   • Upgrade wal-g on all servers and repeat disaster recovery testing with it
   • Explore ZFS local snapshots at the next hardware refresh
   • Make status.matrix.org the source of truth during incidents and keep social media updated

   Lessons

   Most of the difficulty came from the size of the database: restoring 51TB and replaying WALs took far longer than hoped, and incremental backups restore much faster than WAL replay. Running a destructive command on the wrong server was the turning point, and Matrix.org looks for better tooling and processes rather than blaming the individual; working as a trio did not catch the mistake, and server names that suggest a role caused confusion because a primary and a secondary swap roles. The recovery lost no data, but the recovery time needs to be shorter, and the wal-g versions on the disaster recovery server and the production servers had drifted apart.

   View case
3. IL-0280 Kafka brokers in many New Relic cells ran out of disk, leaving US data collection and alerting unusable for hours

   New Relic2021

   Symptoms

   On 29 July 2021, data collection and alerting in New Relic's US region became unusable for many customers for several hours. It began with one unresponsive Kafka broker in a single cell of NRDB, New Relic's telemetry data platform, and over the following hours Kafka brokers in more and more cells failed as their disks filled up.

   Node disk fullThroughput degradationMonitoring outage

   Trigger

   Responders used an incident response tool to extend Kafka data retention for the degraded cell so that its backlog would not be lost. The retention increase was larger than usual, and the tool applied it to all cells of the same type, not only the affected one.

   Root cause

   Operator error

   Component

   Message broker

   Contributing factor

   Harmful manual actionNo staged rolloutSignal buried in noiseUnverified assumption

   Detection

   Engineers saw a Kafka broker in one cell become unresponsive on the morning of 29 July. Later a monitoring alert warned that disk space was running low, but it fired together with unrelated warnings from several other systems and was missed.

   Remediation

   • Restarted the unresponsive Kafka broker by hand, while the automated restart was also running
   • Resolved the disk space problem on the Kafka brokers, after which service recovered slowly

   Lessons

   Cell isolation is the strongest protection against a widespread disruption, and emergency tools have to respect it just as normal operations do. Emergency response tools must be safe to use in chaotic situations without losing their ability to work. Engineers' attention is limited, so incident response processes and alerts need regular review to direct it at the most important signal.

   View case
4. IL-0332 A test run against the production database truncated every table and travis-ci.com was down for about 5.5 hours

   Travis CI2018

   Symptoms

   On 13 March 2018 at 12:14 UTC every table in travis-ci.com's production PostgreSQL database was truncated. Alerts fired right away, and customers reported missing account data; people who signed in saw blank user profiles.

   Data lossService outage

   Trigger

   At 12:04 UTC a developer ran a test suite in an old tmux shell that still had a DATABASEURL environment variable pointing at the production database, and the Database Cleaner gem sent a truncate query to it. The query was blocked for about 10 minutes and executed at 12:14 UTC.

   Root cause

   Operator error

   Component

   Database

   Contributing factor

   No safeguard on destructive changeHarmful manual action

   Detection

   Alerts fired immediately after the truncation, and customers reported missing account data.

   Remediation

   • Took all applications offline
   • Provisioned a replacement database from a snapshot taken that morning
   • Reattached the restored database and brought platform services back up
   • Worked through the backlog of builds until it cleared
   • Revoked all affected login tokens by 14:22 UTC on 14 March
   • Contacted affected customers and advised them to rotate stored credentials and check build logs

   Action items

   • Revoke the truncate permission on the databases
   • Make the internal spechelpers check for the DATABASEURL environment variable
   • Make the developer shell prompt warn when a DATABASE_URL is set
   • Submit a pull request to the Database Cleaner gem guarding against a remote DATABASE_URL
   • Create an alias for the follower database so it is easy to connect to for work against production data
   • Automate database failover and maintenance to cut manual recovery steps

   Lessons

   Developers connected to the production primary with write access because the tooling made the read-only follower hard to reach. While responding, Travis CI left user-facing applications running, which created duplicate user ids, and did not turn some alerts back on, which hid the stopped cron scheduler. Even the most experienced developers can make an inadvertent mistake that causes a large outage. The whole production database was recovered with only about 15 minutes of data loss.

   View case
5. IL-0506 UC Berkeley DataHub logins failed and students got empty home directories after the JupyterHub database was overwritten

   UC Berkeley2017

   Symptoms

   At about 1 AM on 9 February 2017, UC Berkeley's DataHub was reported down: users trying to log in got a proxy error. The hub pod was running, but its log was full of sqlite errors.

   Service outageData lossMisrouted requests

   Trigger

   While debugging, an operator ran `.save jupyterhub.sqlite` in sqlite3 inside the hub pod without having attached the database, which replaced the JupyterHub database file with an empty one.

   Root cause

   Operator error

   Component

   DatabasePersistent volumes and stateful setsJupyterHub

   Contributing factor

   No safeguard on destructive changeUnsafe failure behaviourHarmful manual action

   Detection

   DataHub was reported down at about 1 AM, when users got a proxy error at login. At 1:12 AM a user reported that he could log in but all his notebooks were gone.

   Remediation

   • Scaled the hub to 0 replicas
   • Rebuilt the JupyterHub sqlite database by matching Kubernetes PVC names back to user ids
   • Set cookie_id to null instead of a dummy value and attached the rebuilt database to the hub
   • Changed the cookie secret in the hub pod and restarted it to log all users out
   • Deleted all user pods to log everyone out
   • Ran a script that filled in cookie_id for all users, restarted the hub and deleted all user pods again
   • Dropped the new users with a '.' in their name and renamed the older users from '-' to '.'

   Action items

   • Report upstream that JupyterHub uses only part of the cookie secret and discards the rest
   • Report upstream that the cookie_id column of the users table should have UNIQUE and NOT NULL constraints
   • Ask KubeSpawner to support username hashes in PVC and pod names so the names stay the same when the database is deleted
   • Ask OAuthenticator to support using the id from the Google authenticator as the user id, giving stable user ids
   • Switch to Google Cloud SQL, a hosted and managed MySQL database
   • Take regular and tested backups of the database
   • Start an operational FAQ of things to do and not to do
   • Set up better monitoring and paging
   • Document escalation procedures explicitly

   View case

By root cause1