IL-0217 Honeycomb · 2023-07-25databaseaws
Honeycomb's main MySQL database locked up under cache-miss reads and ran out of connections, a 68-minute total outage
Honeycomb runs several MySQL databases on RDS. A main database holds user accounts, permissions and data schemas; it holds no customer events, which Retriever stores, but the whole system needs it. A shared schema cache protects this database, and ingest traffic keeps that cache warm through last-written timestamps that Retriever writes to the database. When the shared cache is missing, services can run on their own local memory caches for a good while, as long as the database stays up. The switch to the other Retriever cluster on 24 July stopped these timestamp writes because of a feature flag bug: a host told to stop writing never retried, even when the flag was switched back, and only a full reboot fixed it. Frequent deploys that restarted hosts had hidden this flaw, and deployments had been paused during the investigation.
On 25 July the feature flag flip stopped the remaining writes, cache misses sent a surge of reads to the main database, and a deadlock in MySQL's internals, which Honeycomb thinks is non-deterministic and more likely under load, locked its threads until it ran out of connections. From 1:40 p.m. UTC every user-facing component of Honeycomb was down. Responders set up circuit breakers that denied all Shepherd traffic with a 5xx error to protect the database. The database host was hard locked, so they failed over to a replica, which brought the database and the services that depend on it back. They restarted all Retriever hosts, then marked the last-written timestamp of every schema that had seen traffic in the last day as "now", so the schema store reloaded the whole cache; this left incorrect values in the parts of the UI that show how fresh data is. They removed the circuit breaker, restarted query hosts that failed internal checks after the database faults, and Honeycomb was healthy again at 2:48 p.m. UTC.
No new data could be ingested during the outage, and data that users or customers had not buffered for replay was lost for good; the gap stays visible for as long as the data is kept.
- Severity
- –
- Scope
- Total Honeycomb outage of all user-facing components from 1:40 p.m. to 2:48 p.m. UTC: no data could be ingested, processed or accessed, alerting and querying were affected, and data not buffered by senders was lost
- Users affected
- all Honeycomb users
- Region
- –
- Time to detect
- –
- Time to resolve
- 1h 8m
Symptoms
Hours after a routine switch between two clusters of Retriever, Honeycomb's storage and query engine, late on 24 July 2023, the internal SLO for Shepherd, the ingest service, began to burn slowly. The performance was only marginally slow and inconsistent, with no good explanation, so engineers treated it as minor and left it for the morning.
In the morning a chart showed that Retriever's database calls had suddenly stopped. These calls update the records of the last time data arrived for each field or dataset, and the caching mechanism uses those records to decide which dataset schemas to refresh and keep warm for ingest. Without them the ingest cache was undermined. Assuming the new cluster's writes were broken, engineers flipped only the writes back to the other cluster with a feature flag they had used often, but this did not help: because of a bug, a host told to stop writing never tried again, even after the flag was switched back, and only a full reboot moved its writes, so the flip stopped the last writes that were still keeping the system alive.
At 1:40 p.m. UTC on 25 July, just before a command to restart the query engine cluster was ready, the main MySQL database on RDS seized up. With the cache mostly not working, reads on the database had shot up and overloaded it, and a few ordinary writes then seemed to trigger a rare race condition in MySQL's internals that locked thread after thread until operations piled up. The database ran out of connections, services that could not get the data they needed died, and every user-facing component of Honeycomb went down, with no data processed or accessed.
The database host was hard locked: any interaction with a stuck thread waited on the same lock, so it could not be saved in place. Bringing ingest back without a warm cache would have overloaded the database or saturated its connections again. Earlier read-replica failovers had performed poorly, so the team expected a failover of the primary to be costly.
Symptom High latencyDatabase overloadLock contention or deadlockService outage
Root cause
Root cause Software defect
- Trigger
- Late on 24 July Honeycomb switched between two Retriever clusters to avoid a subtle bug, and the new cluster stopped writing the last-written timestamps the schema cache depends on. On the morning of 25 July a feature flag flip meant to move those writes back stopped the remaining writes entirely.
Detection
Hours after the cluster switch, the internal SLO for the ingest service Shepherd started burning slowly. Engineers left it for the morning, when a chart showed that Retriever's database calls updating the last-written records had stopped.
Remediation
- Flipped only the writes back to the other Retriever cluster with a feature flag, which did not fix the problem
- Set up circuit breakers that denied all Shepherd traffic with a 5xx error to protect the database
- Failed the hard-locked database over to a replica
- Restarted all Retriever hosts so they could provide data for the cache
- Set the last-written timestamp of every schema with traffic in the last day to now, so the schema store reloaded the cache
- Removed the circuit breaker and restarted the query hosts that failed internal checks
Action items
- Finish the Retriever cluster migration and remove all code that could disable writes behind the cache
- Explore ways to strengthen the cache further
- Lower the contention put on the database during update storms from new schemas
- Stabilise the performance costs of some operations in the system
- Study instrumentation and experimentation that detect and handle such edge cases better
Lessons
Locally good decisions combined into the outage: the query engine wrote the timestamps because it already saw all the data, the shared cache helped scale ingest and untangle the main database, the cluster switch was done to avoid a bug, frequent deploys hid the feature flag bug, pausing deploys during the investigation kept hosts from restarting, and past near-misses made the team expect a slow failover. Feature flags, frequent deploys, pausing deploys during incidents and learning from near-misses all contributed, although they are among the practices Honeycomb trusts most. For a rare lockup of this kind a database failover is the fastest recovery. Several recent Honeycomb incidents started from trying to avoid a subtle bug.
Sources
Related cases
IL-0388A Google Cloud live migration took a Discord Redis primary offline, and the failover broke API instances and then guildsDiscord · 2017same root cause · shared Database, In-memory cache
IL-0451Setting indexes INVISIBLE on Aurora MySQL 3.10 spiked CPU and ReadIOPS on all three readers but not on the writerDaangn · 2026same root cause · shared Database
IL-0313Decagon voice requests stalled for exactly 300 seconds behind self-managed PgBouncer and exhausted the SQLAlchemy poolDecagon · 2026same root cause · shared Database
IL-0281PostHog's Logs ClickHouse cluster silently deleted almost all its S3 data, losing US logs older than 3 daysPostHog · 2026same root cause · shared Database
IL-0310An Aurora PostgreSQL failover to an upsized reader reversed twice and backend writes failed in read-only transactionsHightouch · 2025same root cause · shared Database
IL-0150A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1Amazon Web Services · 2025same root cause · shared Database
IL-0436A supremum lock left by ROLLBACK TO SAVEPOINT in a long transaction blocked inserts and filled Aurora MySQL connectionsfreee · 2025same root cause · shared Database
IL-0527OpenStreetMap.org made no replication diffs for 16 hours as PostgreSQL logical decoding failed on a memory alloc errorOpenStreetMap Foundation · 2025same root cause · shared Database
IL-0329A routine migration left PGAudit stuck holding Postgres locks, causing intermittent timeouts and a 2-minute outageincident.io · 2025same root cause · shared Database
IL-0376A maintenance update set off needless full syncs between Upstash Redis replicas in eu-west-1, slowing the databasesUpstash · 2025same root cause · shared In-memory cache
IL-0347About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuckScaleway · 2024same root cause · shared Database
IL-0373Heroku Postgres Essential-tier databases in the EU region ran out of memory and refused connections, twice in four daysHeroku · 2024same root cause · shared Database
IL-0024Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three timesHugging Face · 2024same root cause · shared Database
IL-0216Wikimedia on-wiki search was slow or rejected after a datacenter switchover overloaded one Elasticsearch node in codfwWikimedia Foundation · 2023same root cause · shared In-memory cache
IL-0330Requests timed out for two weeks waiting on Go's database/sql connection pool, one for almost 20 secondsincident.io · 2023same root cause · shared Database
IL-0295Pusher's Channels publish API on the mt1 cluster was largely unavailable for 11 minutes while a Redis master failed overPusher · 2022same root cause · shared In-memory cache
IL-0385GitHub MySQL read replicas hit a semaphore deadlock during a schema migration's table rename and kept crash-recoveringGitHub · 2021same root cause · shared Database
IL-0213Roblox was down for 73 hours because key-value writes on its single Consul cluster slowed from under 300 ms to 2 secondsRoblox · 2021same root cause · shared In-memory cache
IL-0462Some MySQL pods on Shopify's GKE clusters took up to 30 minutes to start, slowing instance replacementShopify · 2021same root cause · shared Database
IL-0196Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the dataPalark · 2020same root cause · shared Database
IL-0486Figma's PostgreSQL 9 database on RDS slowed under high write IOPS and lock contention until the site was unavailableFigma · 2020same root cause · shared Database
IL-0241A database shard could not elect a new primary, and the Stripe API failed most requests in two periods on one dayStripe · 2019same root cause · shared Database
IL-0279One failed Redis Cluster replica left only about 1 in 21 calls to Grab's Apollo service succeeding for a minuteGrab · 2019same root cause · shared In-memory cache
IL-0439Database access slowed after each block device and filesystem expansion because the Linux kernel dropped the page cacheCybozu · 2018same root cause · shared Database
IL-0454Baemin billing servers could not open new MariaDB connections at the start of a card-discount eventWoowa Brothers · 2016same root cause · shared Database
IL-0536Amazon SimpleDB in US East returned HTTP 500 to nearly all API calls for two hours after its nodes left the clusterAmazon Web Services · 2014same root cause · shared Database
IL-0485Dropbox went down when an OS upgrade script reinstalled active MySQL master-replica pairs and backup restores ran slowDropbox · 2014same root cause · shared Database
IL-0589Every Kickstarter MySQL replica on Amazon RDS stopped replicating on a duplicate-key error in the backings tableKickstarter · 2013same root cause · shared Database
IL-0303GitHub's MySQL master failed over to a node with a cold buffer pool during a schema migration and GitHub.com went downGitHub · 2012same root cause · shared Database
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0217
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0217/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.